LinkedListTest.TestLoadAndVerify: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: e325588c65a7435aa89e82df0c265bed, Rpc address: host: "127.0.197.130" port: 35633 (Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111))
I20240430 20:00:57.714506   790 webui_checker.cc:88] shutting down CURL thread
W20240430 20:00:57.846307   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240430 20:00:58.186002   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240430 20:00:58.581099  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:58.583949  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:58.586354   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 893
I20240430 20:00:58.597986   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 1155
I20240430 20:00:58.606382   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 802
2024-04-30T20:00:58Z chronyd exiting
I20240430 20:00:58.624158   790 test_util.cc:175] -----------------------------------------------
I20240430 20:00:58.624223   790 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN      ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 20:00:22.131139   790 test_util.cc:268] Using random seed: -772759123
I20240430 20:00:22.137861   790 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 20:00:22.137916   790 linked_list-test.cc:101] --------------
I20240430 20:00:22.137938   790 linked_list-test.cc:102] 50 chains
I20240430 20:00:22.137950   790 linked_list-test.cc:103] 3 tablets
I20240430 20:00:22.137964   790 linked_list-test.cc:104] Mutations on
I20240430 20:00:22.137987   790 linked_list-test.cc:105] --------------
I20240430 20:00:22.138058   790 ts_itest-base.cc:112] Starting cluster with:
I20240430 20:00:22.138083   790 ts_itest-base.cc:113] --------------
I20240430 20:00:22.138103   790 ts_itest-base.cc:114] 3 tablet servers
I20240430 20:00:22.138128   790 ts_itest-base.cc:115] 3 replicas per TS
I20240430 20:00:22.138151   790 ts_itest-base.cc:116] --------------
2024-04-30T20:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T20:00:22Z Disabled control of system clock
I20240430 20:00:22.147872   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.190:40427
--webserver_interface=127.0.197.190
--webserver_port=0
--builtin_ntp_servers=127.0.197.148:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.190:40427
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 20:00:22.250392   802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:22.250633   802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:22.250679   802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:22.253707   802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:00:22.253765   802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:22.253785   802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:00:22.253803   802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:00:22.257334   802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.190:40427
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.190:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.0.197.190
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:22.258198   802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:22.259270   802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:22.266966   810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.267769   808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.268535   807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:22.269366   802 server_base.cc:1034] running on GCE node
I20240430 20:00:22.269850   802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:22.270769   802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:22.271915   802 hybrid_clock.cc:648] HybridClock initialized: now 1714507222271882 us; error 51 us; skew 500 ppm
I20240430 20:00:22.273758   802 webserver.cc:460] Webserver started at http://127.0.197.190:39789/ using document root <none> and password file <none>
I20240430 20:00:22.274302   802 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:22.274360   802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:22.274528   802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:22.276181   802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/instance:
uuid: "bb339ea2531b4852887224d7fa3628c4"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.276647   802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal/instance:
uuid: "bb339ea2531b4852887224d7fa3628c4"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.279945   802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240430 20:00:22.282055   816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.283077   802 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 20:00:22.283205   802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
uuid: "bb339ea2531b4852887224d7fa3628c4"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.283282   802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:22.293898   802 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:22.294477   802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:22.294632   802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:22.300843   802 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.190:40427
I20240430 20:00:22.300848   868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.190:40427 every 8 connection(s)
I20240430 20:00:22.301810   802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/info.pb
I20240430 20:00:22.303136   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 802
I20240430 20:00:22.303313   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal/instance
I20240430 20:00:22.304855   869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.310950   869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4: Bootstrap starting.
I20240430 20:00:22.313724   869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.314435   869 log.cc:824] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:22.316007   869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4: No bootstrap required, opened a new log
I20240430 20:00:22.318959   869 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb339ea2531b4852887224d7fa3628c4" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 40427 } }
I20240430 20:00:22.319221   869 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.319284   869 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb339ea2531b4852887224d7fa3628c4, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.319727   869 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb339ea2531b4852887224d7fa3628c4" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 40427 } }
I20240430 20:00:22.319877   869 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 20:00:22.319945   869 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 20:00:22.320053   869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:22.320793   869 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb339ea2531b4852887224d7fa3628c4" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 40427 } }
I20240430 20:00:22.321090   869 leader_election.cc:302] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb339ea2531b4852887224d7fa3628c4; no voters: 
I20240430 20:00:22.321322   869 leader_election.cc:288] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240430 20:00:22.321650   874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:00:22.321838   874 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [term 1 LEADER]: Becoming Leader. State: Replica: bb339ea2531b4852887224d7fa3628c4, State: Running, Role: LEADER
I20240430 20:00:22.322186   874 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb339ea2531b4852887224d7fa3628c4" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 40427 } }
I20240430 20:00:22.322283   869 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:00:22.323671   876 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb339ea2531b4852887224d7fa3628c4. Latest consensus state: current_term: 1 leader_uuid: "bb339ea2531b4852887224d7fa3628c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb339ea2531b4852887224d7fa3628c4" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 40427 } } }
I20240430 20:00:22.323673   875 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb339ea2531b4852887224d7fa3628c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb339ea2531b4852887224d7fa3628c4" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 40427 } } }
I20240430 20:00:22.323803   876 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [sys.catalog]: This master's current role is: LEADER
I20240430 20:00:22.323859   875 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4 [sys.catalog]: This master's current role is: LEADER
I20240430 20:00:22.324199   887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:00:22.327049   887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:00:22.331794   887 catalog_manager.cc:1347] Generated new cluster ID: c2477dfdfdd242a8b8284e01862aaa83
I20240430 20:00:22.331864   887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:00:22.343997   887 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 20:00:22.344884   887 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:00:22.351528   887 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bb339ea2531b4852887224d7fa3628c4: Generated new TSK 0
I20240430 20:00:22.352165   887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:00:22.356392   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.129:0
--local_ip_for_outbound_sockets=127.0.197.129
--webserver_interface=127.0.197.129
--webserver_port=0
--tserver_master_addrs=127.0.197.190:40427
--builtin_ntp_servers=127.0.197.148:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 20:00:22.464020   893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:22.464228   893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:22.464303   893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:22.467322   893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:22.467403   893 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 20:00:22.467425   893 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 20:00:22.467469   893 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 20:00:22.467527   893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.129
I20240430 20:00:22.471551   893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.0.197.129
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.197.190:40427
--never_fsync=true
--heap_profile_path=/tmp/kudu.893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.129
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:22.472415   893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:22.473513   893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:22.482368   898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.483227   899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.483971   901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:22.484773   893 server_base.cc:1034] running on GCE node
I20240430 20:00:22.485147   893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:22.485702   893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:22.486867   893 hybrid_clock.cc:648] HybridClock initialized: now 1714507222486853 us; error 27 us; skew 500 ppm
I20240430 20:00:22.488765   893 webserver.cc:460] Webserver started at http://127.0.197.129:35117/ using document root <none> and password file <none>
I20240430 20:00:22.489295   893 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:22.489364   893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:22.489544   893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:22.491117   893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/instance:
uuid: "5d585e4520424277ba1ee6643c339315"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.491611   893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal/instance:
uuid: "5d585e4520424277ba1ee6643c339315"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.494714   893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240430 20:00:22.496724   907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.497721   893 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240430 20:00:22.497830   893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
uuid: "5d585e4520424277ba1ee6643c339315"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.497913   893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:22.512511   893 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:22.513100   893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:22.513243   893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:22.513728   893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:00:22.514585   893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:00:22.514632   893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.514686   893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:00:22.514711   893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.523572   893 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.129:43865
I20240430 20:00:22.523622  1020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.129:43865 every 8 connection(s)
I20240430 20:00:22.524500   893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
I20240430 20:00:22.531684   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 893
I20240430 20:00:22.531802   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal/instance
I20240430 20:00:22.533294  1021 heartbeater.cc:347] Connected to a master server at 127.0.197.190:40427
I20240430 20:00:22.533488  1021 heartbeater.cc:464] Registering TS with master...
I20240430 20:00:22.533757   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.130:0
--local_ip_for_outbound_sockets=127.0.197.130
--webserver_interface=127.0.197.130
--webserver_port=0
--tserver_master_addrs=127.0.197.190:40427
--builtin_ntp_servers=127.0.197.148:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 20:00:22.533986  1021 heartbeater.cc:511] Master 127.0.197.190:40427 requested a full tablet report, sending...
I20240430 20:00:22.535326   831 ts_manager.cc:194] Registered new tserver with Master: 5d585e4520424277ba1ee6643c339315 (127.0.197.129:43865)
I20240430 20:00:22.536495   831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.129:54691
W20240430 20:00:22.638893  1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:22.639109  1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:22.639226  1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:22.642236  1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:22.642320  1024 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 20:00:22.642355  1024 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 20:00:22.642410  1024 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 20:00:22.642488  1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.130
I20240430 20:00:22.645989  1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.0.197.130
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.197.190:40427
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.130
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:22.646821  1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:22.647919  1024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:22.656193  1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.657002  1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.657780  1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:22.658584  1024 server_base.cc:1034] running on GCE node
I20240430 20:00:22.658926  1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:22.659482  1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:22.660645  1024 hybrid_clock.cc:648] HybridClock initialized: now 1714507222660633 us; error 26 us; skew 500 ppm
I20240430 20:00:22.662564  1024 webserver.cc:460] Webserver started at http://127.0.197.130:42477/ using document root <none> and password file <none>
I20240430 20:00:22.663285  1024 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:22.663403  1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:22.663604  1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:22.665865  1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/instance:
uuid: "e325588c65a7435aa89e82df0c265bed"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.666309  1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal/instance:
uuid: "e325588c65a7435aa89e82df0c265bed"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.669540  1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20240430 20:00:22.672072  1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.673269  1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240430 20:00:22.673363  1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
uuid: "e325588c65a7435aa89e82df0c265bed"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.673436  1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:22.697712  1024 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:22.698302  1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:22.698446  1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:22.698952  1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:00:22.699823  1024 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:00:22.699870  1024 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.699911  1024 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:00:22.699951  1024 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.709229  1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.130:35633
I20240430 20:00:22.709252  1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.130:35633 every 8 connection(s)
I20240430 20:00:22.710124  1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/info.pb
I20240430 20:00:22.718407  1152 heartbeater.cc:347] Connected to a master server at 127.0.197.190:40427
I20240430 20:00:22.718624  1152 heartbeater.cc:464] Registering TS with master...
I20240430 20:00:22.718967   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 1024
I20240430 20:00:22.719072   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal/instance
I20240430 20:00:22.719200  1152 heartbeater.cc:511] Master 127.0.197.190:40427 requested a full tablet report, sending...
I20240430 20:00:22.720089   831 ts_manager.cc:194] Registered new tserver with Master: e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633)
I20240430 20:00:22.720721   831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.130:48175
I20240430 20:00:22.721372   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.131:0
--local_ip_for_outbound_sockets=127.0.197.131
--webserver_interface=127.0.197.131
--webserver_port=0
--tserver_master_addrs=127.0.197.190:40427
--builtin_ntp_servers=127.0.197.148:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 20:00:22.828584  1155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:22.828823  1155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:22.828907  1155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:22.831995  1155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:22.832091  1155 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 20:00:22.832127  1155 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 20:00:22.832171  1155 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 20:00:22.832248  1155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.131
I20240430 20:00:22.835915  1155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.0.197.131
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.0.197.190:40427
--never_fsync=true
--heap_profile_path=/tmp/kudu.1155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.131
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:22.836838  1155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:22.837944  1155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:22.846642  1160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.847420  1161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:22.848249  1163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:22.849152  1155 server_base.cc:1034] running on GCE node
I20240430 20:00:22.849599  1155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:22.850193  1155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:22.851346  1155 hybrid_clock.cc:648] HybridClock initialized: now 1714507222851324 us; error 37 us; skew 500 ppm
I20240430 20:00:22.853343  1155 webserver.cc:460] Webserver started at http://127.0.197.131:44981/ using document root <none> and password file <none>
I20240430 20:00:22.853924  1155 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:22.853986  1155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:22.854157  1155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:22.855850  1155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/instance:
uuid: "e81650e1870e480893c03a78e6bf7201"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.856338  1155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal/instance:
uuid: "e81650e1870e480893c03a78e6bf7201"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.859632  1155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240430 20:00:22.861752  1169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.862709  1155 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 20:00:22.862810  1155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
uuid: "e81650e1870e480893c03a78e6bf7201"
format_stamp: "Formatted at 2024-04-30 20:00:22 on dist-test-slave-81l2"
I20240430 20:00:22.862890  1155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:22.878072  1155 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:22.878679  1155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:22.878834  1155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:22.879407  1155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:00:22.880275  1155 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:00:22.880323  1155 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.880381  1155 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:00:22.880411  1155 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:22.890044  1155 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.131:42475
I20240430 20:00:22.890067  1282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.131:42475 every 8 connection(s)
I20240430 20:00:22.890949  1155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/info.pb
I20240430 20:00:22.896939   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 1155
I20240430 20:00:22.897073   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal/instance
I20240430 20:00:22.899171  1283 heartbeater.cc:347] Connected to a master server at 127.0.197.190:40427
I20240430 20:00:22.899396  1283 heartbeater.cc:464] Registering TS with master...
I20240430 20:00:22.899914  1283 heartbeater.cc:511] Master 127.0.197.190:40427 requested a full tablet report, sending...
I20240430 20:00:22.900776   833 ts_manager.cc:194] Registered new tserver with Master: e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:22.901365   833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.131:45521
I20240430 20:00:22.911082   790 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:00:22.922174   833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42796:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240430 20:00:22.925107   833 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:00:22.941431  1086 tablet_service.cc:1467] Processing CreateTablet for tablet 03e9b880b7b64b00994e354721a0c2c6 (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 20:00:22.941485  1215 tablet_service.cc:1467] Processing CreateTablet for tablet 134195aa41cd480a97bfe60fca7f3657 (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 20:00:22.941313  1217 tablet_service.cc:1467] Processing CreateTablet for tablet 03e9b880b7b64b00994e354721a0c2c6 (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 20:00:22.941318  1084 tablet_service.cc:1467] Processing CreateTablet for tablet 134195aa41cd480a97bfe60fca7f3657 (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 20:00:22.942348  1216 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb86947a8984490b16a0176fbacf1ac (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 20:00:22.942536  1084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134195aa41cd480a97bfe60fca7f3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.942538  1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e9b880b7b64b00994e354721a0c2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.942864  1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134195aa41cd480a97bfe60fca7f3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.944983   955 tablet_service.cc:1467] Processing CreateTablet for tablet 03e9b880b7b64b00994e354721a0c2c6 (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 20:00:22.946063   955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e9b880b7b64b00994e354721a0c2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.947643  1300 tablet_bootstrap.cc:492] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed: Bootstrap starting.
I20240430 20:00:22.948549  1216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb86947a8984490b16a0176fbacf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.948607  1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e9b880b7b64b00994e354721a0c2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.948885   954 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb86947a8984490b16a0176fbacf1ac (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 20:00:22.949137   954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb86947a8984490b16a0176fbacf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.950505  1300 tablet_bootstrap.cc:654] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.941314  1085 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb86947a8984490b16a0176fbacf1ac (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 20:00:22.950871   953 tablet_service.cc:1467] Processing CreateTablet for tablet 134195aa41cd480a97bfe60fca7f3657 (DEFAULT_TABLE table=TestTable [id=c7815115c9d140708c9739cf49dca348]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 20:00:22.951021  1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb86947a8984490b16a0176fbacf1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.951110   953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134195aa41cd480a97bfe60fca7f3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:22.952173  1300 log.cc:824] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:22.953053  1303 tablet_bootstrap.cc:492] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315: Bootstrap starting.
I20240430 20:00:22.953420  1301 tablet_bootstrap.cc:492] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201: Bootstrap starting.
I20240430 20:00:22.955380  1303 tablet_bootstrap.cc:654] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.955380  1301 tablet_bootstrap.cc:654] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.956465  1301 log.cc:824] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:22.956468  1303 log.cc:824] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:22.957435  1300 tablet_bootstrap.cc:492] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed: No bootstrap required, opened a new log
I20240430 20:00:22.957602  1300 ts_tablet_manager.cc:1367] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240430 20:00:22.957988  1303 tablet_bootstrap.cc:492] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315: No bootstrap required, opened a new log
I20240430 20:00:22.958151  1303 ts_tablet_manager.cc:1367] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240430 20:00:22.958250  1301 tablet_bootstrap.cc:492] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201: No bootstrap required, opened a new log
I20240430 20:00:22.958398  1301 ts_tablet_manager.cc:1367] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240430 20:00:22.960003  1300 raft_consensus.cc:348] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.960175  1300 raft_consensus.cc:374] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.960254  1300 raft_consensus.cc:729] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325588c65a7435aa89e82df0c265bed, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.961231  1300 consensus_queue.cc:260] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.961198  1303 raft_consensus.cc:348] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.961212  1301 raft_consensus.cc:348] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.961385  1301 raft_consensus.cc:374] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.961385  1303 raft_consensus.cc:374] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.961441  1303 raft_consensus.cc:729] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d585e4520424277ba1ee6643c339315, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.961491  1301 raft_consensus.cc:729] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81650e1870e480893c03a78e6bf7201, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.961849  1303 consensus_queue.cc:260] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.961882  1301 consensus_queue.cc:260] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.961980  1300 ts_tablet_manager.cc:1398] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20240430 20:00:22.962343  1152 heartbeater.cc:503] Master 127.0.197.190:40427 was elected leader, sending a full tablet report...
I20240430 20:00:22.962411  1021 heartbeater.cc:503] Master 127.0.197.190:40427 was elected leader, sending a full tablet report...
I20240430 20:00:22.962347  1300 tablet_bootstrap.cc:492] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: Bootstrap starting.
I20240430 20:00:22.963254  1300 tablet_bootstrap.cc:654] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.963316  1283 heartbeater.cc:503] Master 127.0.197.190:40427 was elected leader, sending a full tablet report...
I20240430 20:00:22.963799  1303 ts_tablet_manager.cc:1398] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240430 20:00:22.964103  1303 tablet_bootstrap.cc:492] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315: Bootstrap starting.
I20240430 20:00:22.964849  1301 ts_tablet_manager.cc:1398] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240430 20:00:22.965104  1301 tablet_bootstrap.cc:492] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201: Bootstrap starting.
I20240430 20:00:22.965214  1303 tablet_bootstrap.cc:654] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.966416  1301 tablet_bootstrap.cc:654] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.966671  1303 tablet_bootstrap.cc:492] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315: No bootstrap required, opened a new log
I20240430 20:00:22.966800  1303 ts_tablet_manager.cc:1367] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240430 20:00:22.967700  1301 tablet_bootstrap.cc:492] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201: No bootstrap required, opened a new log
I20240430 20:00:22.967787  1301 ts_tablet_manager.cc:1367] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240430 20:00:22.968340  1301 raft_consensus.cc:348] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.968465  1301 raft_consensus.cc:374] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.968497  1301 raft_consensus.cc:729] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81650e1870e480893c03a78e6bf7201, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.968619  1301 consensus_queue.cc:260] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.968848  1301 ts_tablet_manager.cc:1398] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 20:00:22.968875  1303 raft_consensus.cc:348] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.968966  1303 raft_consensus.cc:374] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.968997  1303 raft_consensus.cc:729] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d585e4520424277ba1ee6643c339315, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.969022  1301 tablet_bootstrap.cc:492] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: Bootstrap starting.
I20240430 20:00:22.969136  1303 consensus_queue.cc:260] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.969393  1303 ts_tablet_manager.cc:1398] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 20:00:22.969575  1303 tablet_bootstrap.cc:492] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: Bootstrap starting.
I20240430 20:00:22.969904  1301 tablet_bootstrap.cc:654] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.970546  1303 tablet_bootstrap.cc:654] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.971196  1301 tablet_bootstrap.cc:492] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: No bootstrap required, opened a new log
I20240430 20:00:22.971335  1301 ts_tablet_manager.cc:1367] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240430 20:00:22.971673  1303 tablet_bootstrap.cc:492] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: No bootstrap required, opened a new log
I20240430 20:00:22.971750  1303 ts_tablet_manager.cc:1367] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240430 20:00:22.971802  1301 raft_consensus.cc:348] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.971915  1301 raft_consensus.cc:374] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.971968  1301 raft_consensus.cc:729] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81650e1870e480893c03a78e6bf7201, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.972090  1301 consensus_queue.cc:260] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.972179  1303 raft_consensus.cc:348] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.972275  1303 raft_consensus.cc:374] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.972307  1303 raft_consensus.cc:729] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d585e4520424277ba1ee6643c339315, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.972350  1301 ts_tablet_manager.cc:1398] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240430 20:00:22.972461  1303 consensus_queue.cc:260] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.973292  1303 ts_tablet_manager.cc:1398] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 20:00:22.974089  1300 tablet_bootstrap.cc:492] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: No bootstrap required, opened a new log
I20240430 20:00:22.974176  1300 ts_tablet_manager.cc:1367] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: Time spent bootstrapping tablet: real 0.012s	user 0.000s	sys 0.002s
I20240430 20:00:22.974670  1300 raft_consensus.cc:348] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.974763  1300 raft_consensus.cc:374] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.974797  1300 raft_consensus.cc:729] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325588c65a7435aa89e82df0c265bed, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.974912  1300 consensus_queue.cc:260] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.975190  1300 ts_tablet_manager.cc:1398] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240430 20:00:22.975340  1300 tablet_bootstrap.cc:492] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed: Bootstrap starting.
I20240430 20:00:22.976162  1300 tablet_bootstrap.cc:654] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:22.977216  1300 tablet_bootstrap.cc:492] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed: No bootstrap required, opened a new log
I20240430 20:00:22.977296  1300 ts_tablet_manager.cc:1367] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20240430 20:00:22.977718  1300 raft_consensus.cc:348] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.977815  1300 raft_consensus.cc:374] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:22.977850  1300 raft_consensus.cc:729] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325588c65a7435aa89e82df0c265bed, State: Initialized, Role: FOLLOWER
I20240430 20:00:22.977978  1300 consensus_queue.cc:260] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:22.978191  1300 ts_tablet_manager.cc:1398] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240430 20:00:23.015434  1306 raft_consensus.cc:482] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:00:23.015973  1306 raft_consensus.cc:504] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.017102  1306 leader_election.cc:288] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d585e4520424277ba1ee6643c339315 (127.0.197.129:43865), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:23.021111  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "e325588c65a7435aa89e82df0c265bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201" is_pre_election: true
I20240430 20:00:23.021108   975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "e325588c65a7435aa89e82df0c265bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d585e4520424277ba1ee6643c339315" is_pre_election: true
I20240430 20:00:23.021389  1237 raft_consensus.cc:2454] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e325588c65a7435aa89e82df0c265bed in term 0.
I20240430 20:00:23.021400   975 raft_consensus.cc:2454] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e325588c65a7435aa89e82df0c265bed in term 0.
I20240430 20:00:23.021824  1041 leader_election.cc:302] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e325588c65a7435aa89e82df0c265bed; no voters: 
I20240430 20:00:23.022081  1306 raft_consensus.cc:2781] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:00:23.022154  1306 raft_consensus.cc:482] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:00:23.022198  1306 raft_consensus.cc:3037] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.023705  1306 raft_consensus.cc:504] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.024061  1306 leader_election.cc:288] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [CANDIDATE]: Term 1 election: Requested vote from peers 5d585e4520424277ba1ee6643c339315 (127.0.197.129:43865), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:23.024405  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "e325588c65a7435aa89e82df0c265bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201"
I20240430 20:00:23.024509  1237 raft_consensus.cc:3037] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.025346  1237 raft_consensus.cc:2454] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e325588c65a7435aa89e82df0c265bed in term 1.
I20240430 20:00:23.025591   975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "e325588c65a7435aa89e82df0c265bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d585e4520424277ba1ee6643c339315"
I20240430 20:00:23.025643  1042 leader_election.cc:302] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e325588c65a7435aa89e82df0c265bed, e81650e1870e480893c03a78e6bf7201; no voters: 
W20240430 20:00:23.025698  1022 tablet.cc:2369] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:00:23.025734   975 raft_consensus.cc:3037] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.025812  1306 raft_consensus.cc:2781] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:00:23.026014  1306 raft_consensus.cc:686] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [term 1 LEADER]: Becoming Leader. State: Replica: e325588c65a7435aa89e82df0c265bed, State: Running, Role: LEADER
I20240430 20:00:23.026363  1306 consensus_queue.cc:237] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.026856   975 raft_consensus.cc:2454] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e325588c65a7435aa89e82df0c265bed in term 1.
I20240430 20:00:23.029992   832 catalog_manager.cc:5484] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed reported cstate change: term changed from 0 to 1, leader changed from <none> to e325588c65a7435aa89e82df0c265bed (127.0.197.130). New cstate: current_term: 1 leader_uuid: "e325588c65a7435aa89e82df0c265bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } health_report { overall_health: UNKNOWN } } }
W20240430 20:00:23.141816  1284 tablet.cc:2369] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:00:23.168398  1308 raft_consensus.cc:482] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:00:23.168551  1308 raft_consensus.cc:504] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.169618  1308 leader_election.cc:288] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:23.173274  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e9b880b7b64b00994e354721a0c2c6" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201" is_pre_election: true
I20240430 20:00:23.173377  1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e9b880b7b64b00994e354721a0c2c6" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325588c65a7435aa89e82df0c265bed" is_pre_election: true
I20240430 20:00:23.173431  1237 raft_consensus.cc:2454] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 0.
I20240430 20:00:23.173578  1106 raft_consensus.cc:2454] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 0.
I20240430 20:00:23.173789   911 leader_election.cc:302] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e81650e1870e480893c03a78e6bf7201; no voters: 
I20240430 20:00:23.174048  1308 raft_consensus.cc:2781] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:00:23.174110  1308 raft_consensus.cc:482] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:00:23.174156  1308 raft_consensus.cc:3037] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.174983  1308 raft_consensus.cc:504] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.175311  1308 leader_election.cc:288] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 election: Requested vote from peers e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:23.175604  1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e9b880b7b64b00994e354721a0c2c6" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325588c65a7435aa89e82df0c265bed"
I20240430 20:00:23.175635  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e9b880b7b64b00994e354721a0c2c6" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201"
I20240430 20:00:23.175695  1106 raft_consensus.cc:3037] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.175740  1237 raft_consensus.cc:3037] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.176494  1237 raft_consensus.cc:2454] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 1.
I20240430 20:00:23.176507  1106 raft_consensus.cc:2454] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 1.
I20240430 20:00:23.176764   910 leader_election.cc:302] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e325588c65a7435aa89e82df0c265bed; no voters: 
I20240430 20:00:23.176930  1308 raft_consensus.cc:2781] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:00:23.177129  1308 raft_consensus.cc:686] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [term 1 LEADER]: Becoming Leader. State: Replica: 5d585e4520424277ba1ee6643c339315, State: Running, Role: LEADER
I20240430 20:00:23.177419  1308 consensus_queue.cc:237] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.180395   832 catalog_manager.cc:5484] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d585e4520424277ba1ee6643c339315 (127.0.197.129). New cstate: current_term: 1 leader_uuid: "5d585e4520424277ba1ee6643c339315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } health_report { overall_health: UNKNOWN } } }
W20240430 20:00:23.211091  1153 tablet.cc:2369] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:00:23.252444  1308 raft_consensus.cc:482] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:00:23.252580  1308 raft_consensus.cc:504] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.253016  1308 leader_election.cc:288] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:23.253523  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86947a8984490b16a0176fbacf1ac" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201" is_pre_election: true
I20240430 20:00:23.253531  1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86947a8984490b16a0176fbacf1ac" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325588c65a7435aa89e82df0c265bed" is_pre_election: true
I20240430 20:00:23.253655  1237 raft_consensus.cc:2454] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 0.
I20240430 20:00:23.253681  1106 raft_consensus.cc:2454] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 0.
I20240430 20:00:23.253966   911 leader_election.cc:302] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e81650e1870e480893c03a78e6bf7201; no voters: 
I20240430 20:00:23.254127  1308 raft_consensus.cc:2781] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:00:23.254182  1308 raft_consensus.cc:482] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:00:23.254209  1308 raft_consensus.cc:3037] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.255044  1308 raft_consensus.cc:504] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.255395  1308 leader_election.cc:288] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 election: Requested vote from peers e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:23.255717  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86947a8984490b16a0176fbacf1ac" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201"
I20240430 20:00:23.255728  1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86947a8984490b16a0176fbacf1ac" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325588c65a7435aa89e82df0c265bed"
I20240430 20:00:23.255811  1237 raft_consensus.cc:3037] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.255834  1106 raft_consensus.cc:3037] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:23.256485  1237 raft_consensus.cc:2454] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 1.
I20240430 20:00:23.256712  1106 raft_consensus.cc:2454] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 1.
I20240430 20:00:23.256773   911 leader_election.cc:302] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e81650e1870e480893c03a78e6bf7201; no voters: 
I20240430 20:00:23.256927  1308 raft_consensus.cc:2781] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:00:23.257025  1308 raft_consensus.cc:686] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [term 1 LEADER]: Becoming Leader. State: Replica: 5d585e4520424277ba1ee6643c339315, State: Running, Role: LEADER
I20240430 20:00:23.257169  1308 consensus_queue.cc:237] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:23.258677   832 catalog_manager.cc:5484] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d585e4520424277ba1ee6643c339315 (127.0.197.129). New cstate: current_term: 1 leader_uuid: "5d585e4520424277ba1ee6643c339315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } health_report { overall_health: UNKNOWN } } }
I20240430 20:00:23.292909   790 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:00:23.294562   790 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5d585e4520424277ba1ee6643c339315 to finish bootstrapping
I20240430 20:00:23.300274   790 ts_itest-base.cc:233] Waiting for 3 tablets on tserver e325588c65a7435aa89e82df0c265bed to finish bootstrapping
I20240430 20:00:23.307817   790 ts_itest-base.cc:233] Waiting for 3 tablets on tserver e81650e1870e480893c03a78e6bf7201 to finish bootstrapping
I20240430 20:00:23.314095  1324 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
  http://127.0.197.190:39789/mem-trackers
  http://127.0.197.131:44981/tablets
  http://127.0.197.129:35117/metrics
  http://127.0.197.190:39789/masters
  http://127.0.197.130:42477/mem-trackers
  http://127.0.197.129:35117/stacks
  http://127.0.197.129:35117/threadz?group=all
  http://127.0.197.131:44981/transactions?tablet_id=134195aa41cd480a97bfe60fca7f3657
  http://127.0.197.190:39789/startup
  http://127.0.197.129:35117/tablets
  http://127.0.197.190:39789/ipki-ca-cert
  http://127.0.197.130:42477/threadz
  http://127.0.197.129:35117/mem-trackers
  http://127.0.197.129:35117/maintenance-manager
  http://127.0.197.190:39789/tablet-servers
  http://127.0.197.130:42477/transactions?tablet_id=134195aa41cd480a97bfe60fca7f3657
  http://127.0.197.131:44981/mem-trackers
  http://127.0.197.130:42477/startup
  http://127.0.197.190:39789/ipki-ca-cert-pem
  http://127.0.197.131:44981/metrics
  http://127.0.197.129:35117/startup
  http://127.0.197.190:39789/stacks
  http://127.0.197.129:35117/transactions?tablet_id=134195aa41cd480a97bfe60fca7f3657
  http://127.0.197.131:44981/stacks
  http://127.0.197.190:39789/metrics
  http://127.0.197.131:44981/threadz
  http://127.0.197.190:39789/dump-entities
  http://127.0.197.131:44981/threadz?group=all
  http://127.0.197.129:35117/threadz
  http://127.0.197.130:42477/maintenance-manager
  http://127.0.197.190:39789/tables
  http://127.0.197.190:39789/ipki-ca-cert-der
  http://127.0.197.190:39789/threadz?group=all
  http://127.0.197.190:39789/threadz
  http://127.0.197.130:42477/scans
  http://127.0.197.130:42477/tablets
  http://127.0.197.130:42477/stacks
  http://127.0.197.130:42477/metrics
  http://127.0.197.131:44981/startup
  http://127.0.197.130:42477/threadz?group=all
  http://127.0.197.129:35117/scans
  http://127.0.197.131:44981/maintenance-manager
  http://127.0.197.131:44981/scans
I20240430 20:00:23.317215   790 linked_list-test-util.h:401] Running for: 30.000s
I20240430 20:00:23.317282   790 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 20:00:23.317305   790 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240430 20:00:23.337432  1237 raft_consensus.cc:1261] T 9cb86947a8984490b16a0176fbacf1ac P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Refusing update from remote peer 5d585e4520424277ba1ee6643c339315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:00:23.338135   975 raft_consensus.cc:1261] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Refusing update from remote peer e325588c65a7435aa89e82df0c265bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:00:23.338157  1106 raft_consensus.cc:1261] T 9cb86947a8984490b16a0176fbacf1ac P e325588c65a7435aa89e82df0c265bed [term 1 FOLLOWER]: Refusing update from remote peer 5d585e4520424277ba1ee6643c339315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:00:23.338718  1237 raft_consensus.cc:1261] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Refusing update from remote peer e325588c65a7435aa89e82df0c265bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:00:23.338656  1316 consensus_queue.cc:1035] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:23.338824  1320 consensus_queue.cc:1035] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:23.339411  1320 consensus_queue.cc:1035] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:23.339541  1237 raft_consensus.cc:1261] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Refusing update from remote peer 5d585e4520424277ba1ee6643c339315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:00:23.339951  1106 raft_consensus.cc:1261] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed [term 1 FOLLOWER]: Refusing update from remote peer 5d585e4520424277ba1ee6643c339315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:00:23.340221  1320 consensus_queue.cc:1035] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:23.340402  1308 consensus_queue.cc:1035] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:23.341178  1306 consensus_queue.cc:1035] T 134195aa41cd480a97bfe60fca7f3657 P e325588c65a7435aa89e82df0c265bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:23.349550  1332 mvcc.cc:205] Tried to move back new op lower bound from 7022621586782773248 to 7022621586462650368. Current Snapshot: MvccSnapshot[applied={T|T < 7022621586782773248}]
I20240430 20:00:23.351281  1340 mvcc.cc:205] Tried to move back new op lower bound from 7022621586786349056 to 7022621585519570944. Current Snapshot: MvccSnapshot[applied={T|T < 7022621586786349056}]
I20240430 20:00:23.351646  1346 mvcc.cc:205] Tried to move back new op lower bound from 7022621586786349056 to 7022621585519570944. Current Snapshot: MvccSnapshot[applied={T|T < 7022621586786349056}]
I20240430 20:00:25.427232  1350 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:26.263783  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:26.267289  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:26.649664  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:26.718348  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:26.782840  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:27.162446  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.444s	user 0.270s	sys 0.000s Metrics: {"bytes_written":143266,"cfile_init":1,"compiler_manager_pool.queue_time_us":44040,"dirs.queue_time_us":11288,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":7922,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2918,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":7863,"thread_start_us":158,"threads_started":2}
I20240430 20:00:27.164024  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 136159 bytes on disk
I20240430 20:00:27.166826  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240430 20:00:27.331585   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.548s	user 0.263s	sys 0.004s Metrics: {"bytes_written":144956,"cfile_init":1,"compiler_manager_pool.queue_time_us":18309,"dirs.queue_time_us":21515,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":2719,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1288,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7962,"thread_start_us":183,"threads_started":2}
I20240430 20:00:27.332940  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 137278 bytes on disk
I20240430 20:00:27.336232   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240430 20:00:27.351126  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.701s	user 0.242s	sys 0.012s Metrics: {"bytes_written":141200,"cfile_init":1,"compiler_manager_pool.queue_time_us":4961,"compiler_manager_pool.run_cpu_time_us":246150,"compiler_manager_pool.run_wall_time_us":272632,"dirs.queue_time_us":9144,"dirs.run_cpu_time_us":396,"dirs.run_wall_time_us":5725,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17296,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":7733,"spinlock_wait_cycles":225024,"thread_start_us":175,"threads_started":2}
I20240430 20:00:27.352363  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 134684 bytes on disk
I20240430 20:00:27.353029  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240430 20:00:28.780089  1350 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:29.541323  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:29.544302  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:30.608424  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:30.673394  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:30.844416  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:31.343999   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.499s	user 0.244s	sys 0.000s Metrics: {"bytes_written":153558,"cfile_init":1,"dirs.queue_time_us":17030,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":962,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3181,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8452,"thread_start_us":76,"threads_started":1}
I20240430 20:00:31.345192  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014816
I20240430 20:00:31.579524  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.900s	user 0.387s	sys 0.000s Metrics: {"bytes_written":15980,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":52710,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1554,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1699,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8261,"thread_start_us":84,"threads_started":1,"update_count":2482}
I20240430 20:00:31.581009  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014821
I20240430 20:00:31.660385  1350 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:31.810504  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.202s	user 0.276s	sys 0.003s Metrics: {"bytes_written":15982,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":23238,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":937,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6251,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8324,"thread_start_us":102,"threads_started":1,"update_count":2482}
I20240430 20:00:31.812568  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014821
I20240430 20:00:32.334030   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.984s	user 0.501s	sys 0.004s Metrics: {"bytes_written":2478,"cfile_cache_hit":13,"cfile_cache_hit_bytes":143959,"cfile_cache_miss":57,"cfile_cache_miss_bytes":939397,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":23640,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":9425,"drs_written":1,"lbm_read_time_us":2016,"lbm_reads_lt_1ms":89,"lbm_write_time_us":2514,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":523724,"reinsert_count":0,"rows_written":16414,"thread_start_us":80,"threads_started":1,"update_count":357}
I20240430 20:00:32.334900  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.009717
I20240430 20:00:32.439419   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.104s	user 0.052s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3589,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":112,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":548,"lbm_read_time_us":452,"lbm_reads_lt_1ms":24,"lbm_write_time_us":278,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":357}
I20240430 20:00:32.440105  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 300745 bytes on disk
I20240430 20:00:32.440557   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240430 20:00:32.548833  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.735s	user 0.392s	sys 0.008s Metrics: {"bytes_written":289699,"cfile_cache_hit":14,"cfile_cache_hit_bytes":140600,"cfile_cache_miss":58,"cfile_cache_miss_bytes":917746,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":11080,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":5915,"drs_written":1,"lbm_read_time_us":1876,"lbm_reads_lt_1ms":94,"lbm_write_time_us":2346,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":522280,"rows_written":16057,"thread_start_us":492,"threads_started":8}
I20240430 20:00:32.549706  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 296875 bytes on disk
I20240430 20:00:32.550155  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240430 20:00:32.574885  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.993s	user 0.479s	sys 0.008s Metrics: {"bytes_written":618,"cfile_cache_hit":14,"cfile_cache_hit_bytes":141554,"cfile_cache_miss":58,"cfile_cache_miss_bytes":918794,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":17837,"dirs.run_cpu_time_us":910,"dirs.run_wall_time_us":13081,"drs_written":1,"lbm_read_time_us":2151,"lbm_reads_lt_1ms":94,"lbm_write_time_us":2889,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":520980,"reinsert_count":0,"rows_written":16124,"thread_start_us":7926,"threads_started":5,"update_count":67}
I20240430 20:00:32.575599  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.002475
I20240430 20:00:32.614176  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:32.616914  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:32.618176  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.037s	user 0.027s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":876,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":90,"lbm_read_time_us":302,"lbm_reads_lt_1ms":24,"lbm_write_time_us":210,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":67}
I20240430 20:00:32.618602  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 297564 bytes on disk
I20240430 20:00:32.618988  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240430 20:00:33.317754   790 linked_list-test-util.h:417] Sample at HT timestamp: 7022621627636580352 Inserted count: 30800
I20240430 20:00:34.720260  1350 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:34.946281  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:35.056906  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:35.123605  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:35.340066   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.392s	user 0.232s	sys 0.002s Metrics: {"bytes_written":146432,"cfile_init":1,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1449,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7929,"thread_start_us":74,"threads_started":1}
I20240430 20:00:35.340740  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling LogGCOp(03e9b880b7b64b00994e354721a0c2c6): free 1048472 bytes of WAL
I20240430 20:00:35.340975   912 log_reader.cc:386] T 03e9b880b7b64b00994e354721a0c2c6: removed 1 log segments from log reader
I20240430 20:00:35.341051   912 log.cc:1076] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315: Deleting log segment in path: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal/wals/03e9b880b7b64b00994e354721a0c2c6/wal-000000001 (ops 1-729)
I20240430 20:00:35.341540   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: LogGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240430 20:00:35.341864  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 137560 bytes on disk
W20240430 20:00:35.346897  1326 meta_cache.cc:1261] Time spent looking up entry by key: real 0.084s	user 0.003s	sys 0.000s
I20240430 20:00:35.347565   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240430 20:00:35.354319  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014704
I20240430 20:00:35.582578  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:35.585619  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:35.669406  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.546s	user 0.241s	sys 0.000s Metrics: {"bytes_written":161616,"cfile_init":1,"dirs.queue_time_us":7890,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1204,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2235,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":8626,"spinlock_wait_cycles":1280,"thread_start_us":79,"threads_started":1}
I20240430 20:00:35.670225  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling LogGCOp(03e9b880b7b64b00994e354721a0c2c6): free 1050509 bytes of WAL
I20240430 20:00:35.670526  1043 log_reader.cc:386] T 03e9b880b7b64b00994e354721a0c2c6: removed 1 log segments from log reader
I20240430 20:00:35.670663  1043 log.cc:1076] T 03e9b880b7b64b00994e354721a0c2c6 P e325588c65a7435aa89e82df0c265bed: Deleting log segment in path: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal/wals/03e9b880b7b64b00994e354721a0c2c6/wal-000000001 (ops 1-734)
I20240430 20:00:35.676167  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: LogGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.006s	user 0.001s	sys 0.000s Metrics: {}
I20240430 20:00:35.676540  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014698
I20240430 20:00:35.696614  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.640s	user 0.236s	sys 0.003s Metrics: {"bytes_written":160552,"cfile_init":1,"dirs.queue_time_us":23460,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":922,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1532,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":8561,"thread_start_us":68,"threads_started":1}
I20240430 20:00:35.709509  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling LogGCOp(03e9b880b7b64b00994e354721a0c2c6): free 1047543 bytes of WAL
I20240430 20:00:35.711405  1174 log_reader.cc:386] T 03e9b880b7b64b00994e354721a0c2c6: removed 1 log segments from log reader
I20240430 20:00:35.711490  1174 log.cc:1076] T 03e9b880b7b64b00994e354721a0c2c6 P e81650e1870e480893c03a78e6bf7201: Deleting log segment in path: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal/wals/03e9b880b7b64b00994e354721a0c2c6/wal-000000001 (ops 1-729)
I20240430 20:00:35.711973  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: LogGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20240430 20:00:35.715600  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014699
I20240430 20:00:36.685259   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.331s	user 0.553s	sys 0.012s Metrics: {"bytes_written":3387,"cfile_cache_hit":14,"cfile_cache_hit_bytes":236204,"cfile_cache_miss":76,"cfile_cache_miss_bytes":1402782,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1286,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":4993,"drs_written":1,"lbm_read_time_us":2320,"lbm_reads_lt_1ms":108,"lbm_write_time_us":3161,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":796345,"reinsert_count":0,"rows_written":24343,"thread_start_us":447,"threads_started":8,"update_count":498}
I20240430 20:00:36.685985  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.008949
I20240430 20:00:36.828044   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.142s	user 0.067s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":5031,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":54,"lbm_read_time_us":348,"lbm_reads_lt_1ms":24,"lbm_write_time_us":2192,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":7,"reinsert_count":0,"update_count":498}
I20240430 20:00:36.832710  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 455396 bytes on disk
I20240430 20:00:36.833171   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240430 20:00:36.960644  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.245s	user 0.573s	sys 0.004s Metrics: {"bytes_written":5143,"cfile_cache_hit":12,"cfile_cache_hit_bytes":143506,"cfile_cache_miss":76,"cfile_cache_miss_bytes":1511326,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":2682,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":22787,"drs_written":1,"lbm_read_time_us":2118,"lbm_reads_lt_1ms":108,"lbm_write_time_us":4399,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":800643,"reinsert_count":0,"rows_written":24618,"spinlock_wait_cycles":7424,"thread_start_us":1048,"threads_started":7,"update_count":773}
I20240430 20:00:36.962297  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 458263 bytes on disk
I20240430 20:00:36.963449  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240430 20:00:36.963950  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.013421
I20240430 20:00:37.054260  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.090s	user 0.039s	sys 0.008s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7303,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":226,"lbm_reads_lt_1ms":20,"lbm_write_time_us":202,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":773}
I20240430 20:00:37.058393  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.382s	user 0.565s	sys 0.008s Metrics: {"bytes_written":5974,"cfile_cache_hit":12,"cfile_cache_hit_bytes":51490,"cfile_cache_miss":80,"cfile_cache_miss_bytes":1609650,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":57560,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":62125,"drs_written":1,"lbm_read_time_us":2574,"lbm_reads_lt_1ms":116,"lbm_write_time_us":6205,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":67,"num_input_rowsets":2,"peak_mem_usage":802677,"reinsert_count":0,"rows_written":24750,"thread_start_us":431,"threads_started":8,"update_count":905}
I20240430 20:00:37.059180  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.015497
I20240430 20:00:37.104245  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.045s	user 0.044s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":8381,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":66,"lbm_read_time_us":263,"lbm_reads_lt_1ms":24,"lbm_write_time_us":223,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":905}
I20240430 20:00:37.104874  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 459773 bytes on disk
I20240430 20:00:37.105361  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240430 20:00:37.884404  1350 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:38.539557  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:38.542342  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:38.557891  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(9cb86947a8984490b16a0176fbacf1ac): perf score=1.000000
I20240430 20:00:38.587741  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(9cb86947a8984490b16a0176fbacf1ac): perf score=1.000000
I20240430 20:00:38.608244  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling FlushMRSOp(9cb86947a8984490b16a0176fbacf1ac): perf score=1.000000
I20240430 20:00:38.914317  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(9cb86947a8984490b16a0176fbacf1ac) complete. Timing: real 0.356s	user 0.221s	sys 0.002s Metrics: {"bytes_written":139894,"cfile_init":1,"dirs.queue_time_us":12336,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":790,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1598,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7642,"thread_start_us":85,"threads_started":1}
I20240430 20:00:38.915089  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(134195aa41cd480a97bfe60fca7f3657): perf score=1.000000
I20240430 20:00:39.069095   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(9cb86947a8984490b16a0176fbacf1ac) complete. Timing: real 0.481s	user 0.254s	sys 0.008s Metrics: {"bytes_written":139894,"cfile_init":1,"dirs.queue_time_us":11796,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":5967,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1416,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7642,"thread_start_us":86,"threads_started":1}
I20240430 20:00:39.069859  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(134195aa41cd480a97bfe60fca7f3657): perf score=1.000000
I20240430 20:00:39.162667  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: FlushMRSOp(9cb86947a8984490b16a0176fbacf1ac) complete. Timing: real 0.554s	user 0.222s	sys 0.004s Metrics: {"bytes_written":139998,"cfile_init":1,"dirs.queue_time_us":26064,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1753,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1348,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7651,"thread_start_us":75,"threads_started":1}
I20240430 20:00:39.163390  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling FlushMRSOp(134195aa41cd480a97bfe60fca7f3657): perf score=1.000000
I20240430 20:00:39.470465  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.554s	user 0.220s	sys 0.000s Metrics: {"bytes_written":2743,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":584,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":810,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1484,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7687,"update_count":397}
I20240430 20:00:39.480301  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling MajorDeltaCompactionOp(134195aa41cd480a97bfe60fca7f3657): perf score=0.022857
I20240430 20:00:39.507140  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: MajorDeltaCompactionOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.020s	user 0.017s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3596,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":43,"lbm_read_time_us":277,"lbm_reads_lt_1ms":24,"lbm_write_time_us":232,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":78,"threads_started":1,"update_count":397}
I20240430 20:00:39.507797  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(134195aa41cd480a97bfe60fca7f3657): 143671 bytes on disk
I20240430 20:00:39.508337  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240430 20:00:39.508905  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(9cb86947a8984490b16a0176fbacf1ac): 143451 bytes on disk
I20240430 20:00:39.509258  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(9cb86947a8984490b16a0176fbacf1ac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240430 20:00:39.627188  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: FlushMRSOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.463s	user 0.299s	sys 0.000s Metrics: {"bytes_written":140602,"cfile_init":1,"dirs.queue_time_us":324,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1514,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2152,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7743}
I20240430 20:00:39.628057  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(134195aa41cd480a97bfe60fca7f3657): 147978 bytes on disk
I20240430 20:00:39.631330  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240430 20:00:39.631918  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(9cb86947a8984490b16a0176fbacf1ac): 143514 bytes on disk
I20240430 20:00:39.633069  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(9cb86947a8984490b16a0176fbacf1ac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240430 20:00:39.654631   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.585s	user 0.223s	sys 0.000s Metrics: {"bytes_written":3067,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6396,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":2290,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13015,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7738,"thread_start_us":67,"threads_started":1,"update_count":448}
I20240430 20:00:39.655365  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling MajorDeltaCompactionOp(134195aa41cd480a97bfe60fca7f3657): perf score=0.025372
I20240430 20:00:39.678107   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: MajorDeltaCompactionOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.022s	user 0.020s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3954,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":37,"lbm_read_time_us":422,"lbm_reads_lt_1ms":24,"lbm_write_time_us":237,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":89,"threads_started":1,"update_count":448}
I20240430 20:00:39.678776  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(134195aa41cd480a97bfe60fca7f3657): 144294 bytes on disk
I20240430 20:00:39.679523   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(134195aa41cd480a97bfe60fca7f3657) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240430 20:00:39.680018  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(9cb86947a8984490b16a0176fbacf1ac): 143451 bytes on disk
I20240430 20:00:39.680366   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(9cb86947a8984490b16a0176fbacf1ac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240430 20:00:39.931272  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:40.010910  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:40.135854  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:40.236830   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.305s	user 0.211s	sys 0.000s Metrics: {"bytes_written":147944,"cfile_init":1,"dirs.queue_time_us":13850,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":927,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1668,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8046,"thread_start_us":76,"threads_started":1}
I20240430 20:00:40.237690  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014573
I20240430 20:00:40.590386  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.454s	user 0.279s	sys 0.000s Metrics: {"bytes_written":10680,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9227,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":7284,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1565,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7981,"thread_start_us":75,"threads_started":1,"update_count":1647}
I20240430 20:00:40.594206  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 128869 bytes on disk
I20240430 20:00:40.598074  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240430 20:00:40.598738  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014569
I20240430 20:00:40.656152  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.645s	user 0.266s	sys 0.004s Metrics: {"bytes_written":9392,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1597,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":6488,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1544,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7908,"thread_start_us":81,"threads_started":1,"update_count":1442}
I20240430 20:00:40.657505  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 129295 bytes on disk
I20240430 20:00:40.658005  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240430 20:00:40.658763  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014571
I20240430 20:00:41.739497  1350 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:41.753116  1397 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48354 (ReqId={client: 6f11892c35fc4bf68416e8fb3fa09790, seq_no=2956, attempt_no=0}) took 1570 ms. Trace:
I20240430 20:00:41.753319  1397 rpcz_store.cc:276] 0430 20:00:40.183135 (+     0us) service_pool.cc:168] Inserting onto call queue
0430 20:00:40.183194 (+    59us) service_pool.cc:225] Handling call
0430 20:00:41.753099 (+1569905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:00:40.183548 (+     0us) write_op.cc:183] PREPARE: starting on tablet 03e9b880b7b64b00994e354721a0c2c6
0430 20:00:40.183650 (+   102us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:00:40.183654 (+     4us) write_op.cc:427] Acquired schema lock
0430 20:00:40.183656 (+     2us) tablet.cc:660] Decoding operations
0430 20:00:40.188434 (+  4778us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:00:40.188444 (+    10us) write_op.cc:633] Partition lock acquired for write op
0430 20:00:40.188448 (+     4us) tablet.cc:683] Acquiring locks for 1647 operations
0430 20:00:40.233820 (+ 45372us) tablet.cc:699] Row locks acquired
0430 20:00:40.233822 (+     2us) write_op.cc:260] PREPARE: finished
0430 20:00:40.233852 (+    30us) write_op.cc:270] Start()
0430 20:00:40.233868 (+    16us) write_op.cc:276] Timestamp: P: 1714507240233840 usec, L: 0
0430 20:00:40.233869 (+     1us) op_driver.cc:331] REPLICATION: starting
0430 20:00:40.233965 (+    96us) log.cc:842] Serialized 18467 byte log entry
0430 20:00:40.236367 (+  2402us) op_driver.cc:444] REPLICATION: finished
0430 20:00:40.249629 (+ 13262us) write_op.cc:301] APPLY: starting
0430 20:00:40.249641 (+    12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:00:40.830673 (+581032us) tablet.cc:1368] finished BulkCheckPresence
0430 20:00:40.830684 (+    11us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:00:41.737902 (+907218us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:00:41.738577 (+   675us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=4599,key_file_lookups=2610,delta_file_lookups=0,mrs_lookups=342
0430 20:00:41.738582 (+     5us) write_op.cc:312] APPLY: finished
0430 20:00:41.743577 (+  4995us) log.cc:842] Serialized 12512 byte log entry
0430 20:00:41.752461 (+  8884us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:00:41.752938 (+   477us) write_op.cc:446] Released schema lock
0430 20:00:41.753033 (+    95us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13220,"cfile_cache_hit":6291,"cfile_cache_hit_bytes":127984810,"cfile_cache_miss":2,"cfile_cache_miss_bytes":47368,"lbm_read_time_us":53,"lbm_reads_lt_1ms":2,"num_ops":1647,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":11334,"prepare.run_wall_time_us":50497,"raft.queue_time_us":851,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":314,"replication_time_us":2484,"spinlock_wait_cycles":768}]]}
I20240430 20:00:42.147468  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.489s	user 0.834s	sys 0.010s Metrics: {"bytes_written":584886,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1043,"cfile_cache_miss":98,"cfile_cache_miss_bytes":2214607,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":16006,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":2524,"drs_written":1,"lbm_read_time_us":3547,"lbm_reads_lt_1ms":134,"lbm_write_time_us":6496,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":77,"num_input_rowsets":2,"peak_mem_usage":1079360,"rows_written":32526,"thread_start_us":643,"threads_started":8}
I20240430 20:00:42.148387  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 621749 bytes on disk
I20240430 20:00:42.149735  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240430 20:00:42.323287   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 2.080s	user 0.763s	sys 0.008s Metrics: {"bytes_written":8491,"cfile_cache_hit":14,"cfile_cache_hit_bytes":324907,"cfile_cache_miss":92,"cfile_cache_miss_bytes":1864763,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":30359,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":3157,"lbm_reads_lt_1ms":128,"lbm_write_time_us":17845,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":79,"num_input_rowsets":2,"peak_mem_usage":1051354,"reinsert_count":0,"rows_written":32389,"thread_start_us":477,"threads_started":8,"update_count":1305}
I20240430 20:00:42.326176  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 607613 bytes on disk
I20240430 20:00:42.328359   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240430 20:00:42.329041  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.016882
I20240430 20:00:42.407390   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.078s	user 0.068s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":11775,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":453,"lbm_read_time_us":629,"lbm_reads_lt_1ms":20,"lbm_write_time_us":317,"lbm_writes_lt_1ms":8,"mutex_wait_us":370,"reinsert_count":0,"update_count":1305}
I20240430 20:00:42.415977  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.817s	user 0.987s	sys 0.012s Metrics: {"bytes_written":588619,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1043,"cfile_cache_miss":98,"cfile_cache_miss_bytes":2228225,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":8206,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":612,"drs_written":1,"lbm_read_time_us":3471,"lbm_reads_lt_1ms":134,"lbm_write_time_us":4412,"lbm_writes_lt_1ms":78,"num_input_rowsets":2,"peak_mem_usage":1086242,"rows_written":32731,"spinlock_wait_cycles":1280,"thread_start_us":501,"threads_started":8}
I20240430 20:00:42.416914  1153 maintenance_manager.cc:419] P e325588c65a7435aa89e82df0c265bed: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 625791 bytes on disk
I20240430 20:00:42.417493  1043 maintenance_manager.cc:643] P e325588c65a7435aa89e82df0c265bed: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240430 20:00:42.691731  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:42.695293  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:43.322830   790 linked_list-test-util.h:417] Sample at HT timestamp: 7022621668613668864 Inserted count: 57350
F20240430 20:00:43.463174  1350 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
    @     0x7fa11fe16dcd  google::LogMessage::Fail() at ??:0
    @     0x7fa11fe1ab93  google::LogMessage::SendToLog() at ??:0
    @     0x7fa11fe167cc  google::LogMessage::Flush() at ??:0
    @     0x7fa11fe17f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fa1211f85b1  kudu::operator-() at ??:0
    @     0x7fa1232dbfd2  kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7fa1232c505b  kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7fa1232c4f90  kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7fa12329b9bd  kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7fa123ec785d  kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7fa123ec6936  _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
    @     0x7fa123ecf145  _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
    @     0x7fa11ed91d09  std::function<>::operator()() at ??:0
    @     0x7fa11ed8c107  kudu::Webserver::RunPathHandler() at ??:0
    @     0x7fa11ed8ba38  kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7fa11ed8b123  kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7fa11edb3b89  handle_request at ??:0
    @     0x7fa11edb626b  worker_thread at ??:0
    @     0x7fa121a956db  start_thread at ??:0
    @     0x7fa11efcd71f  clone at ??:0
W20240430 20:00:43.695446  1288 connection.cc:535] client connection to 127.0.197.130:35633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:00:43.695648  1288 meta_cache.cc:302] tablet 134195aa41cd480a97bfe60fca7f3657: replica e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:00:43.696075   910 connection.cc:535] client connection to 127.0.197.130:35633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:00:43.696223   910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:00:43.697230   910 consensus_peers.cc:485] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 20:00:43.697317   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 20:00:43.700067   934 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.701257   934 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.711370  1196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.711373  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.724370   934 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.731642  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.736801  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.747807   934 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.755244   934 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.767347  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.769349  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.793773   934 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.793773   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.806432  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.821833   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.837287  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.840423  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.855952   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.875535   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.896970   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.898229  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.918984  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.942862   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.963603  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.965622  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:43.988307   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.012533   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.037858   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.040396  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.065017  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.098242   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.124276   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.125572  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.153017  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.187395   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.220289  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.220289   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.255877  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.287300  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.323434   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.324512   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.361425  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.397818   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.434183  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.436198  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.474274   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.514233   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.556035  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.556097  1196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
I20240430 20:00:44.558410  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:44.560178  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:00:44.596776   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.641384   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.684792  1196 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.684819  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.730823   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.732822  1288 meta_cache.cc:302] tablet 134195aa41cd480a97bfe60fca7f3657: replica e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633) has failed: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111)
W20240430 20:00:44.777204   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.823089  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.824054  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.869864   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.918254   935 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48354: Illegal state: replica 5d585e4520424277ba1ee6643c339315 is not leader of this config: current role FOLLOWER
I20240430 20:00:44.957047  1558 raft_consensus.cc:482] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e325588c65a7435aa89e82df0c265bed)
I20240430 20:00:44.957171  1558 raft_consensus.cc:504] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:44.958251  1558 leader_election.cc:288] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d585e4520424277ba1ee6643c339315 (127.0.197.129:43865), e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633)
W20240430 20:00:44.959193  1172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111)
W20240430 20:00:44.960033  1172 leader_election.cc:334] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111)
I20240430 20:00:44.962873   975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "e81650e1870e480893c03a78e6bf7201" candidate_term: 2 candidate_status { last_received { term: 1 index: 1151 } } ignore_live_leader: false dest_uuid: "5d585e4520424277ba1ee6643c339315" is_pre_election: true
I20240430 20:00:44.963311  1172 leader_election.cc:302] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e81650e1870e480893c03a78e6bf7201; no voters: 5d585e4520424277ba1ee6643c339315, e325588c65a7435aa89e82df0c265bed
I20240430 20:00:44.963522  1558 raft_consensus.cc:2726] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:00:44.964526  1526 raft_consensus.cc:482] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e325588c65a7435aa89e82df0c265bed)
I20240430 20:00:44.964618  1526 raft_consensus.cc:504] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:44.964983  1526 leader_election.cc:288] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:44.965302  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 2 candidate_status { last_received { term: 1 index: 1151 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201" is_pre_election: true
I20240430 20:00:44.965421  1237 raft_consensus.cc:2454] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 1.
I20240430 20:00:44.965718   911 leader_election.cc:302] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e81650e1870e480893c03a78e6bf7201; no voters: 
I20240430 20:00:44.965876  1526 raft_consensus.cc:2781] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:00:44.965926  1526 raft_consensus.cc:482] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e325588c65a7435aa89e82df0c265bed)
I20240430 20:00:44.965968  1526 raft_consensus.cc:3037] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:00:44.966153   910 leader_election.cc:334] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111)
I20240430 20:00:44.966909  1526 raft_consensus.cc:504] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:44.967247  1526 leader_election.cc:288] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 2 election: Requested vote from peers e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633), e81650e1870e480893c03a78e6bf7201 (127.0.197.131:42475)
I20240430 20:00:44.968041  1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134195aa41cd480a97bfe60fca7f3657" candidate_uuid: "5d585e4520424277ba1ee6643c339315" candidate_term: 2 candidate_status { last_received { term: 1 index: 1151 } } ignore_live_leader: false dest_uuid: "e81650e1870e480893c03a78e6bf7201"
I20240430 20:00:44.968147  1237 raft_consensus.cc:3037] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:00:44.968175  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
I20240430 20:00:44.969177  1237 raft_consensus.cc:2454] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d585e4520424277ba1ee6643c339315 in term 2.
W20240430 20:00:44.969249  1197 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53438: Illegal state: replica e81650e1870e480893c03a78e6bf7201 is not leader of this config: current role FOLLOWER
W20240430 20:00:44.969339   910 leader_election.cc:334] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111)
I20240430 20:00:44.969607   911 leader_election.cc:302] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d585e4520424277ba1ee6643c339315, e81650e1870e480893c03a78e6bf7201; no voters: e325588c65a7435aa89e82df0c265bed
I20240430 20:00:44.969799  1526 raft_consensus.cc:2781] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:00:44.969954  1526 raft_consensus.cc:686] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [term 2 LEADER]: Becoming Leader. State: Replica: 5d585e4520424277ba1ee6643c339315, State: Running, Role: LEADER
I20240430 20:00:44.970105  1526 consensus_queue.cc:237] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1151, Committed index: 1151, Last appended: 1.1151, Last appended by leader: 1151, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } }
I20240430 20:00:44.971776   826 catalog_manager.cc:5484] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 reported cstate change: term changed from 1 to 2, leader changed from e325588c65a7435aa89e82df0c265bed (127.0.197.130) to 5d585e4520424277ba1ee6643c339315 (127.0.197.129). New cstate: current_term: 2 leader_uuid: "5d585e4520424277ba1ee6643c339315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d585e4520424277ba1ee6643c339315" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325588c65a7435aa89e82df0c265bed" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 35633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 } health_report { overall_health: UNKNOWN } } }
I20240430 20:00:45.019454  1237 raft_consensus.cc:1261] T 134195aa41cd480a97bfe60fca7f3657 P e81650e1870e480893c03a78e6bf7201 [term 2 FOLLOWER]: Refusing update from remote peer 5d585e4520424277ba1ee6643c339315: Log matching property violated. Preceding OpId in replica: term: 1 index: 1151. Preceding OpId from leader: term: 2 index: 1153. (index mismatch)
I20240430 20:00:45.019830  1526 consensus_queue.cc:1035] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81650e1870e480893c03a78e6bf7201" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1152, Last known committed idx: 1151, Time since last communication: 0.000s
W20240430 20:00:45.020218   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 20:00:45.847473   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:00:46.101743  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:46.106815  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:46.157488  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:46.165815  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
W20240430 20:00:46.335997   910 consensus_peers.cc:485] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:00:46.528421   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.362s	user 0.226s	sys 0.004s Metrics: {"bytes_written":147162,"cfile_init":1,"dirs.queue_time_us":14056,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1099,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6415,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":8064,"thread_start_us":88,"threads_started":1}
I20240430 20:00:46.529194  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014455
I20240430 20:00:46.582103  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.424s	user 0.378s	sys 0.000s Metrics: {"bytes_written":145095,"cfile_init":1,"dirs.queue_time_us":19181,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":841,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1725,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7927,"thread_start_us":87,"threads_started":1}
I20240430 20:00:46.582990  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014453
W20240430 20:00:47.173615   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:00:47.605240   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.072s	user 0.915s	sys 0.014s Metrics: {"bytes_written":2308,"cfile_cache_hit":12,"cfile_cache_hit_bytes":48585,"cfile_cache_miss":110,"cfile_cache_miss_bytes":2715717,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1194,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":7331,"drs_written":1,"lbm_read_time_us":3737,"lbm_reads_lt_1ms":146,"lbm_write_time_us":5275,"lbm_writes_lt_1ms":97,"num_input_rowsets":2,"peak_mem_usage":1339109,"reinsert_count":0,"rows_written":40453,"spinlock_wait_cycles":1920,"thread_start_us":431,"threads_started":7,"update_count":325}
I20240430 20:00:47.606076  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 774898 bytes on disk
I20240430 20:00:47.606535   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240430 20:00:47.607031  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.003673
I20240430 20:00:47.691431   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.084s	user 0.060s	sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3606,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":264,"lbm_reads_lt_1ms":20,"lbm_write_time_us":519,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":325}
I20240430 20:00:48.163772  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:48.166440  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:00:48.436810   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 20:00:48.587430  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.998s	user 1.550s	sys 0.000s Metrics: {"bytes_written":2308,"cfile_cache_hit":12,"cfile_cache_hit_bytes":230883,"cfile_cache_miss":108,"cfile_cache_miss_bytes":2541427,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":2018,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2749,"drs_written":1,"lbm_read_time_us":4708,"lbm_reads_lt_1ms":140,"lbm_write_time_us":10260,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":95,"num_input_rowsets":2,"peak_mem_usage":1339157,"reinsert_count":0,"rows_written":40453,"thread_start_us":597,"threads_started":7,"update_count":325}
I20240430 20:00:48.588310  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 774898 bytes on disk
I20240430 20:00:48.588735  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240430 20:00:48.589322  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.003673
W20240430 20:00:48.651517   910 consensus_peers.cc:485] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 20:00:48.691351  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.102s	user 0.096s	sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":3606,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":424,"lbm_read_time_us":362,"lbm_reads_lt_1ms":20,"lbm_write_time_us":371,"lbm_writes_lt_1ms":8,"mutex_wait_us":301,"reinsert_count":0,"update_count":325}
W20240430 20:00:48.961059   910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240430 20:00:49.896808   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 20:00:49.943997  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:49.945586  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.000000
I20240430 20:00:50.148077  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:50.158320  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:50.177264  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.233s	user 0.206s	sys 0.000s Metrics: {"bytes_written":151600,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1382,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8266,"thread_start_us":78,"threads_started":1}
I20240430 20:00:50.177948  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014456
I20240430 20:00:50.403805   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: FlushMRSOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.457s	user 0.245s	sys 0.008s Metrics: {"bytes_written":15123,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":32802,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":728,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1813,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8266,"thread_start_us":74,"threads_started":1,"update_count":2346}
I20240430 20:00:50.404695  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 126987 bytes on disk
I20240430 20:00:50.416810   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240430 20:00:50.417593  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6): perf score=1.014453
W20240430 20:00:50.797606   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 20:00:51.317663   910 consensus_peers.cc:485] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 20:00:52.149686  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.959s	user 1.471s	sys 0.008s Metrics: {"bytes_written":15108,"cfile_cache_hit":14,"cfile_cache_hit_bytes":415954,"cfile_cache_miss":132,"cfile_cache_miss_bytes":2906370,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":4134,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1330,"drs_written":1,"lbm_read_time_us":4559,"lbm_reads_lt_1ms":168,"lbm_write_time_us":7228,"lbm_writes_lt_1ms":110,"num_input_rowsets":2,"peak_mem_usage":1579755,"reinsert_count":0,"rows_written":48719,"thread_start_us":373,"threads_started":7,"update_count":2346}
I20240430 20:00:52.150521  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6): perf score=0.019905
I20240430 20:00:52.247804  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: MajorDeltaCompactionOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.097s	user 0.093s	sys 0.001s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":20599,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":163,"lbm_read_time_us":320,"lbm_reads_lt_1ms":25,"lbm_write_time_us":248,"lbm_writes_lt_1ms":9,"mutex_wait_us":93,"reinsert_count":0,"update_count":2346}
I20240430 20:00:52.248463  1284 maintenance_manager.cc:419] P e81650e1870e480893c03a78e6bf7201: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 918988 bytes on disk
I20240430 20:00:52.252406  1174 maintenance_manager.cc:643] P e81650e1870e480893c03a78e6bf7201: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240430 20:00:52.315997   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 20:00:52.362217   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: CompactRowSetsOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 1.943s	user 1.114s	sys 0.008s Metrics: {"bytes_written":872938,"cfile_cache_hit":14,"cfile_cache_hit_bytes":367445,"cfile_cache_miss":138,"cfile_cache_miss_bytes":2984919,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":1174,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":10623,"drs_written":1,"lbm_read_time_us":4734,"lbm_reads_lt_1ms":174,"lbm_write_time_us":14813,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":102,"num_input_rowsets":2,"peak_mem_usage":1621837,"rows_written":48719,"thread_start_us":500,"threads_started":8}
I20240430 20:00:52.363487  1022 maintenance_manager.cc:419] P 5d585e4520424277ba1ee6643c339315: Scheduling UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6): 937923 bytes on disk
I20240430 20:00:52.364967   912 maintenance_manager.cc:643] P 5d585e4520424277ba1ee6643c339315: UndoDeltaBlockGCOp(03e9b880b7b64b00994e354721a0c2c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240430 20:00:52.650267  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:52.652318  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:00:53.335592   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 20:00:53.349320   790 linked_list-test-util.h:417] Sample at HT timestamp: 7022621709679763456 Inserted count: 80850
I20240430 20:00:53.349411   790 linked_list-test-util.h:421] Finished inserting list. Added 80850 in chain
I20240430 20:00:53.349428   790 linked_list-test-util.h:422] Last entries inserted had keys:
I20240430 20:00:53.349447   790 linked_list-test-util.h:424] 0: -3844083645271638016
I20240430 20:00:53.349470   790 linked_list-test-util.h:424] 1: 782572754005196801
I20240430 20:00:53.349484   790 linked_list-test-util.h:424] 2: 5409229153282031618
I20240430 20:00:53.349498   790 linked_list-test-util.h:424] 3: -8410999258638974973
I20240430 20:00:53.349516   790 linked_list-test-util.h:424] 4: -7628145029657067516
I20240430 20:00:53.349537   790 linked_list-test-util.h:424] 5: -3001629367868522491
I20240430 20:00:53.349552   790 linked_list-test-util.h:424] 6: 1625027031408312326
I20240430 20:00:53.349566   790 linked_list-test-util.h:424] 7: 6251683430685147143
I20240430 20:00:53.349579   790 linked_list-test-util.h:424] 8: 7034396922178764808
I20240430 20:00:53.349594   790 linked_list-test-util.h:424] 9: -6785690752253951991
I20240430 20:00:53.349612   790 linked_list-test-util.h:424] 10: -2159175090465406966
I20240430 20:00:53.349637   790 linked_list-test-util.h:424] 11: 2467481308811427851
I20240430 20:00:53.349651   790 linked_list-test-util.h:424] 12: 3250194800305045516
I20240430 20:00:53.349665   790 linked_list-test-util.h:424] 13: 7876851199581880333
I20240430 20:00:53.349679   790 linked_list-test-util.h:424] 14: -5943236474850836466
I20240430 20:00:53.349694   790 linked_list-test-util.h:424] 15: -1316439338085580785
I20240430 20:00:53.349707   790 linked_list-test-util.h:424] 16: -533866584080383984
I20240430 20:00:53.349726   790 linked_list-test-util.h:424] 17: 4092649077708161041
I20240430 20:00:53.349741   790 linked_list-test-util.h:424] 18: 8719305476984995858
I20240430 20:00:53.349756   790 linked_list-test-util.h:424] 19: -5100500722471010285
I20240430 20:00:53.349776   790 linked_list-test-util.h:424] 20: -4318068705954103276
I20240430 20:00:53.349795   790 linked_list-test-util.h:424] 21: 308587693322731541
I20240430 20:00:53.349814   790 linked_list-test-util.h:424] 22: 4935103355111276566
I20240430 20:00:53.349828   790 linked_list-test-util.h:424] 23: -8884702844344729577
I20240430 20:00:53.349843   790 linked_list-test-util.h:424] 24: -8102130090339532776
I20240430 20:00:53.349857   790 linked_list-test-util.h:424] 25: -3475614428550987751
I20240430 20:00:53.349871   790 linked_list-test-util.h:424] 26: 1151323445702557722
I20240430 20:00:53.349889   790 linked_list-test-util.h:424] 27: 1933755462219464731
I20240430 20:00:53.349905   790 linked_list-test-util.h:424] 28: 6560411861496299548
I20240430 20:00:53.349928   790 linked_list-test-util.h:424] 29: -7259675812936417251
I20240430 20:00:53.349944   790 linked_list-test-util.h:424] 30: -2632878676171161570
I20240430 20:00:53.349967   790 linked_list-test-util.h:424] 31: -1850305922165964769
I20240430 20:00:53.349990   790 linked_list-test-util.h:424] 32: 2776209739622580256
I20240430 20:00:53.350021   790 linked_list-test-util.h:424] 33: 7402866138899415073
I20240430 20:00:53.350046   790 linked_list-test-util.h:424] 34: -6416940060556591070
I20240430 20:00:53.350069   790 linked_list-test-util.h:424] 35: -5634508044039684061
I20240430 20:00:53.350091   790 linked_list-test-util.h:424] 36: -1007851644762849244
I20240430 20:00:53.350114   790 linked_list-test-util.h:424] 37: 3618945492002406437
I20240430 20:00:53.350142   790 linked_list-test-util.h:424] 38: 8245601891279241254
I20240430 20:00:53.350165   790 linked_list-test-util.h:424] 39: 9028033907796148263
I20240430 20:00:53.350189   790 linked_list-test-util.h:424] 40: -4792053766636568536
I20240430 20:00:53.350216   790 linked_list-test-util.h:424] 41: -165115892383023063
I20240430 20:00:53.350239   790 linked_list-test-util.h:424] 42: 4461399769405521962
I20240430 20:00:53.350263   790 linked_list-test-util.h:424] 43: 5243972523410718763
I20240430 20:00:53.350291   790 linked_list-test-util.h:424] 44: -8576255888510287828
I20240430 20:00:53.350315   790 linked_list-test-util.h:424] 45: -3949318014256742355
I20240430 20:00:53.350345   790 linked_list-test-util.h:424] 46: 677338385020092462
I20240430 20:00:53.350373   790 linked_list-test-util.h:424] 47: 1459770401536999471
I20240430 20:00:53.350395   790 linked_list-test-util.h:424] 48: 6086708275790544944
I20240430 20:00:53.350419   790 linked_list-test-util.h:424] 49: -7733520136130461647
I20240430 20:00:53.389106   790 linked_list-test-util.h:509] Verifying Snapshot: P: 1714507233309712 usec, L: 1 Expected Rows: 30800
I20240430 20:00:53.416524   790 linked_list-test-util.h:528] Calling callback on tserver 5d585e4520424277ba1ee6643c339315
I20240430 20:00:53.817945   790 meta_cache.cc:1510] marking tablet server e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633) as failed
W20240430 20:00:53.818069   790 meta_cache.cc:302] tablet 03e9b880b7b64b00994e354721a0c2c6: replica e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111)
W20240430 20:00:53.921614   910 consensus_peers.cc:485] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 20:00:54.022053   790 linked_list-test-util.h:767] Done collecting results (30800 rows in 605.697ms)
I20240430 20:00:54.057478   790 linked_list-test-util.h:556] Snapshot: P: 1714507233309712 usec, L: 1 verified. Result: OK
I20240430 20:00:54.059940   790 linked_list-test-util.h:509] Verifying Snapshot: P: 1714507243313884 usec, L: 1 Expected Rows: 57350
I20240430 20:00:54.068341   790 linked_list-test-util.h:528] Calling callback on tserver e81650e1870e480893c03a78e6bf7201
W20240430 20:00:54.207317   910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240430 20:00:54.211906  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:54.215268  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:54.736613   790 linked_list-test-util.h:767] Done collecting results (57350 rows in 668.346ms)
I20240430 20:00:54.787665   790 linked_list-test-util.h:556] Snapshot: P: 1714507243313884 usec, L: 1 verified. Result: OK
I20240430 20:00:54.789846   790 linked_list-test-util.h:509] Verifying Snapshot: P: 1714507253339786 usec, L: 1 Expected Rows: 80850
I20240430 20:00:54.795689   790 linked_list-test-util.h:528] Calling callback on tserver e81650e1870e480893c03a78e6bf7201
W20240430 20:00:55.083267   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 20:00:55.733891  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:55.736575  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 20:00:55.781774   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 20:00:56.223551   790 linked_list-test-util.h:767] Done collecting results (80850 rows in 1427.95ms)
I20240430 20:00:56.298705   790 linked_list-test-util.h:556] Snapshot: P: 1714507253339786 usec, L: 1 verified. Result: OK
I20240430 20:00:56.301316   790 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 80850
W20240430 20:00:56.478374   910 consensus_peers.cc:485] T 9cb86947a8984490b16a0176fbacf1ac P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 20:00:57.196038  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:57.199390  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:57.596940   790 linked_list-test-util.h:767] Done collecting results (80850 rows in 1287.36ms)
I20240430 20:00:57.696398   790 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240430 20:00:57.696794   790 linked_list-test-util.h:692] Successfully verified 80850 rows
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: e325588c65a7435aa89e82df0c265bed, Rpc address: host: "127.0.197.130" port: 35633 (Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111))
I20240430 20:00:57.714506   790 webui_checker.cc:88] shutting down CURL thread
W20240430 20:00:57.846307   910 consensus_peers.cc:485] T 134195aa41cd480a97bfe60fca7f3657 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240430 20:00:58.186002   910 consensus_peers.cc:485] T 03e9b880b7b64b00994e354721a0c2c6 P 5d585e4520424277ba1ee6643c339315 -> Peer e325588c65a7435aa89e82df0c265bed (127.0.197.130:35633): Couldn't send request to peer e325588c65a7435aa89e82df0c265bed. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.130:35633: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240430 20:00:58.581099  1330 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:58.583949  1327 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 20:00:58.586354   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 893
I20240430 20:00:58.597986   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 1155
I20240430 20:00:58.606382   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 802
2024-04-30T20:00:58Z chronyd exiting
I20240430 20:00:58.624158   790 test_util.cc:175] -----------------------------------------------
I20240430 20:00:58.624223   790 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714507222116962-790-0
[  FAILED  ] LinkedListTest.TestLoadAndVerify (36495 ms)
[ RUN      ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240430 20:00:58.624886   790 test_util.cc:268] Using random seed: -736265359
I20240430 20:00:58.625586   790 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 20:00:58.625633   790 linked_list-test.cc:101] --------------
I20240430 20:00:58.625650   790 linked_list-test.cc:102] 50 chains
I20240430 20:00:58.625666   790 linked_list-test.cc:103] 3 tablets
I20240430 20:00:58.625694   790 linked_list-test.cc:104] Mutations on
I20240430 20:00:58.625725   790 linked_list-test.cc:105] --------------
I20240430 20:00:58.625854   790 ts_itest-base.cc:112] Starting cluster with:
I20240430 20:00:58.625914   790 ts_itest-base.cc:113] --------------
I20240430 20:00:58.625939   790 ts_itest-base.cc:114] 3 tablet servers
I20240430 20:00:58.625962   790 ts_itest-base.cc:115] 3 replicas per TS
I20240430 20:00:58.625989   790 ts_itest-base.cc:116] --------------
2024-04-30T20:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T20:00:58Z Disabled control of system clock
I20240430 20:00:58.634441   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.190:45189
--webserver_interface=127.0.197.190
--webserver_port=0
--builtin_ntp_servers=127.0.197.148:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.190:45189
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 20:00:58.739886  1686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:58.740111  1686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:58.740156  1686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:58.743098  1686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:00:58.743147  1686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:58.743198  1686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:00:58.743230  1686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:00:58.746934  1686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.190:45189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.190:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.0.197.190
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.1686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:58.747886  1686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:58.749017  1686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:58.757017  1691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:58.757978  1686 server_base.cc:1034] running on GCE node
W20240430 20:00:58.758419  1694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:58.759285  1692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:58.759791  1686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:58.760756  1686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:58.761906  1686 hybrid_clock.cc:648] HybridClock initialized: now 1714507258761891 us; error 31 us; skew 500 ppm
I20240430 20:00:58.763862  1686 webserver.cc:460] Webserver started at http://127.0.197.190:45585/ using document root <none> and password file <none>
I20240430 20:00:58.764423  1686 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:58.764487  1686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:58.764660  1686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:58.766337  1686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/instance:
uuid: "31b892d1c9e6445e824cf49ad1d7724c"
format_stamp: "Formatted at 2024-04-30 20:00:58 on dist-test-slave-81l2"
I20240430 20:00:58.766815  1686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal/instance:
uuid: "31b892d1c9e6445e824cf49ad1d7724c"
format_stamp: "Formatted at 2024-04-30 20:00:58 on dist-test-slave-81l2"
I20240430 20:00:58.770114  1686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240430 20:00:58.772316  1700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:58.773289  1686 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240430 20:00:58.773411  1686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
uuid: "31b892d1c9e6445e824cf49ad1d7724c"
format_stamp: "Formatted at 2024-04-30 20:00:58 on dist-test-slave-81l2"
I20240430 20:00:58.773495  1686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:58.806345  1686 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:58.807037  1686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:58.807228  1686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:58.813764  1686 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.190:45189
I20240430 20:00:58.813776  1752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.190:45189 every 8 connection(s)
I20240430 20:00:58.814759  1686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/data/info.pb
I20240430 20:00:58.817358  1753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:58.819736   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 1686
I20240430 20:00:58.819898   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/master-0/wal/instance
I20240430 20:00:58.822925  1753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c: Bootstrap starting.
I20240430 20:00:58.825028  1753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:58.825714  1753 log.cc:824] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:58.827332  1753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c: No bootstrap required, opened a new log
I20240430 20:00:58.829965  1753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b892d1c9e6445e824cf49ad1d7724c" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 45189 } }
I20240430 20:00:58.830132  1753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:58.830185  1753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b892d1c9e6445e824cf49ad1d7724c, State: Initialized, Role: FOLLOWER
I20240430 20:00:58.830528  1753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b892d1c9e6445e824cf49ad1d7724c" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 45189 } }
I20240430 20:00:58.830626  1753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 20:00:58.830667  1753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 20:00:58.830746  1753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:58.831462  1753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b892d1c9e6445e824cf49ad1d7724c" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 45189 } }
I20240430 20:00:58.831745  1753 leader_election.cc:302] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b892d1c9e6445e824cf49ad1d7724c; no voters: 
I20240430 20:00:58.831960  1753 leader_election.cc:288] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240430 20:00:58.832074  1758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:00:58.832334  1758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [term 1 LEADER]: Becoming Leader. State: Replica: 31b892d1c9e6445e824cf49ad1d7724c, State: Running, Role: LEADER
I20240430 20:00:58.832626  1758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b892d1c9e6445e824cf49ad1d7724c" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 45189 } }
I20240430 20:00:58.833134  1753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:00:58.834439  1760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b892d1c9e6445e824cf49ad1d7724c. Latest consensus state: current_term: 1 leader_uuid: "31b892d1c9e6445e824cf49ad1d7724c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b892d1c9e6445e824cf49ad1d7724c" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 45189 } } }
I20240430 20:00:58.834549  1760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [sys.catalog]: This master's current role is: LEADER
I20240430 20:00:58.834908  1759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b892d1c9e6445e824cf49ad1d7724c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b892d1c9e6445e824cf49ad1d7724c" member_type: VOTER last_known_addr { host: "127.0.197.190" port: 45189 } } }
I20240430 20:00:58.834987  1759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c [sys.catalog]: This master's current role is: LEADER
W20240430 20:00:58.837566  1773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 20:00:58.837627  1773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 20:00:58.837730  1768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:00:58.838433  1768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:00:58.842782  1768 catalog_manager.cc:1347] Generated new cluster ID: 2cf9f6dd821d46a791c4e5adedd39da6
I20240430 20:00:58.842849  1768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:00:58.863492  1768 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 20:00:58.864734  1768 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:00:58.872089  1768 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 31b892d1c9e6445e824cf49ad1d7724c: Generated new TSK 0
I20240430 20:00:58.872931  1768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:00:58.883200   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.129:0
--local_ip_for_outbound_sockets=127.0.197.129
--webserver_interface=127.0.197.129
--webserver_port=0
--tserver_master_addrs=127.0.197.190:45189
--builtin_ntp_servers=127.0.197.148:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 20:00:58.988301  1777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:58.988525  1777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:58.988606  1777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:58.991626  1777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:58.991768  1777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.129
I20240430 20:00:58.995317  1777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.0.197.129
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.0.197.190:45189
--never_fsync=true
--heap_profile_path=/tmp/kudu.1777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.129
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:58.996177  1777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:58.997267  1777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:59.005584  1783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:59.006531  1785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:59.007334  1782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:59.008150  1777 server_base.cc:1034] running on GCE node
I20240430 20:00:59.008543  1777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:59.009115  1777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:59.010285  1777 hybrid_clock.cc:648] HybridClock initialized: now 1714507259010270 us; error 36 us; skew 500 ppm
I20240430 20:00:59.012182  1777 webserver.cc:460] Webserver started at http://127.0.197.129:37753/ using document root <none> and password file <none>
I20240430 20:00:59.012743  1777 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:59.012832  1777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:59.013028  1777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:59.014667  1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/instance:
uuid: "416de875ac014d1fbd60b90f51f626d2"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.015141  1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal/instance:
uuid: "416de875ac014d1fbd60b90f51f626d2"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.018496  1777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.001s
I20240430 20:00:59.020756  1791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.021881  1777 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20240430 20:00:59.022001  1777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
uuid: "416de875ac014d1fbd60b90f51f626d2"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.022085  1777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:59.034729  1777 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:59.035442  1777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:59.035599  1777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:59.036123  1777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:00:59.037009  1777 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:00:59.037060  1777 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.037120  1777 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:00:59.037148  1777 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.045606  1777 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.129:36663
I20240430 20:00:59.045630  1904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.129:36663 every 8 connection(s)
I20240430 20:00:59.046523  1777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
I20240430 20:00:59.049109   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 1777
I20240430 20:00:59.049252   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal/instance
I20240430 20:00:59.051735   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.130:0
--local_ip_for_outbound_sockets=127.0.197.130
--webserver_interface=127.0.197.130
--webserver_port=0
--tserver_master_addrs=127.0.197.190:45189
--builtin_ntp_servers=127.0.197.148:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240430 20:00:59.055969  1905 heartbeater.cc:347] Connected to a master server at 127.0.197.190:45189
I20240430 20:00:59.056183  1905 heartbeater.cc:464] Registering TS with master...
I20240430 20:00:59.056700  1905 heartbeater.cc:511] Master 127.0.197.190:45189 requested a full tablet report, sending...
I20240430 20:00:59.058130  1717 ts_manager.cc:194] Registered new tserver with Master: 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663)
I20240430 20:00:59.059309  1717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.129:34193
W20240430 20:00:59.160607  1908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:59.160866  1908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:59.160957  1908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:59.163906  1908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:59.164047  1908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.130
I20240430 20:00:59.167572  1908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.0.197.130
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.0.197.190:45189
--never_fsync=true
--heap_profile_path=/tmp/kudu.1908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.130
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:59.168476  1908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:59.169559  1908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:59.177762  1914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:59.178597  1913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:59.179469  1916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:59.180364  1908 server_base.cc:1034] running on GCE node
I20240430 20:00:59.180721  1908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:59.181260  1908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:59.182411  1908 hybrid_clock.cc:648] HybridClock initialized: now 1714507259182402 us; error 27 us; skew 500 ppm
I20240430 20:00:59.184253  1908 webserver.cc:460] Webserver started at http://127.0.197.130:34739/ using document root <none> and password file <none>
I20240430 20:00:59.184798  1908 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:59.184859  1908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:59.185025  1908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:59.186600  1908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/instance:
uuid: "e563869b282a4389abf74929d93a015f"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.187084  1908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal/instance:
uuid: "e563869b282a4389abf74929d93a015f"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.190203  1908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20240430 20:00:59.192315  1922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.193341  1908 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240430 20:00:59.193462  1908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
uuid: "e563869b282a4389abf74929d93a015f"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.193549  1908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:59.207300  1908 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:59.207913  1908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:59.208063  1908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:59.208571  1908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:00:59.209420  1908 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:00:59.209470  1908 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.209528  1908 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:00:59.209555  1908 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.217942  1908 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.130:41937
I20240430 20:00:59.217972  2035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.130:41937 every 8 connection(s)
I20240430 20:00:59.218842  1908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/data/info.pb
I20240430 20:00:59.227452   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 1908
I20240430 20:00:59.227583   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-1/wal/instance
I20240430 20:00:59.227627  2036 heartbeater.cc:347] Connected to a master server at 127.0.197.190:45189
I20240430 20:00:59.227847  2036 heartbeater.cc:464] Registering TS with master...
I20240430 20:00:59.228394  2036 heartbeater.cc:511] Master 127.0.197.190:45189 requested a full tablet report, sending...
I20240430 20:00:59.229316  1717 ts_manager.cc:194] Registered new tserver with Master: e563869b282a4389abf74929d93a015f (127.0.197.130:41937)
I20240430 20:00:59.229656   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.131:0
--local_ip_for_outbound_sockets=127.0.197.131
--webserver_interface=127.0.197.131
--webserver_port=0
--tserver_master_addrs=127.0.197.190:45189
--builtin_ntp_servers=127.0.197.148:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240430 20:00:59.229962  1717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.130:40105
W20240430 20:00:59.338100  2039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:00:59.338335  2039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:00:59.338388  2039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:00:59.341475  2039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:00:59.341586  2039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.131
I20240430 20:00:59.345206  2039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.0.197.131
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.0.197.190:45189
--never_fsync=true
--heap_profile_path=/tmp/kudu.2039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.131
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:00:59.346081  2039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:00:59.347205  2039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:00:59.355319  2044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:59.356145  2045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:00:59.356959  2047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:00:59.357894  2039 server_base.cc:1034] running on GCE node
I20240430 20:00:59.358269  2039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:00:59.358824  2039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:00:59.359977  2039 hybrid_clock.cc:648] HybridClock initialized: now 1714507259359964 us; error 28 us; skew 500 ppm
I20240430 20:00:59.361887  2039 webserver.cc:460] Webserver started at http://127.0.197.131:39281/ using document root <none> and password file <none>
I20240430 20:00:59.362449  2039 fs_manager.cc:359] Metadata directory not provided
I20240430 20:00:59.362507  2039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:00:59.362692  2039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:00:59.364390  2039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/instance:
uuid: "f3ba6f48197d4bf09cdc2af16980c15c"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.364902  2039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal/instance:
uuid: "f3ba6f48197d4bf09cdc2af16980c15c"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.368221  2039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240430 20:00:59.370297  2053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.371290  2039 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240430 20:00:59.371402  2039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
uuid: "f3ba6f48197d4bf09cdc2af16980c15c"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:00:59.371485  2039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:00:59.392581  2039 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:00:59.393267  2039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:00:59.393441  2039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:00:59.393978  2039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:00:59.394867  2039 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:00:59.394918  2039 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.394979  2039 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:00:59.395009  2039 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:00:59.403368  2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.131:34131 every 8 connection(s)
I20240430 20:00:59.403352  2039 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.131:34131
I20240430 20:00:59.404318  2039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/data/info.pb
I20240430 20:00:59.405373   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 2039
I20240430 20:00:59.405508   790 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-2/wal/instance
I20240430 20:00:59.413079  2167 heartbeater.cc:347] Connected to a master server at 127.0.197.190:45189
I20240430 20:00:59.413271  2167 heartbeater.cc:464] Registering TS with master...
I20240430 20:00:59.413741  2167 heartbeater.cc:511] Master 127.0.197.190:45189 requested a full tablet report, sending...
I20240430 20:00:59.414657  1717 ts_manager.cc:194] Registered new tserver with Master: f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131)
I20240430 20:00:59.415149  1717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.131:48701
I20240430 20:00:59.419392   790 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:00:59.427735  1717 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49064:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240430 20:00:59.430310  1717 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:00:59.445888  2101 tablet_service.cc:1467] Processing CreateTablet for tablet baab1c5f96484c18a9aebb43ab1f520d (DEFAULT_TABLE table=TestTable [id=20f60bf97ff640c4aac9153a754124c8]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 20:00:59.445888  1839 tablet_service.cc:1467] Processing CreateTablet for tablet baab1c5f96484c18a9aebb43ab1f520d (DEFAULT_TABLE table=TestTable [id=20f60bf97ff640c4aac9153a754124c8]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 20:00:59.446545  1970 tablet_service.cc:1467] Processing CreateTablet for tablet baab1c5f96484c18a9aebb43ab1f520d (DEFAULT_TABLE table=TestTable [id=20f60bf97ff640c4aac9153a754124c8]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 20:00:59.446894  1839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baab1c5f96484c18a9aebb43ab1f520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:59.446894  2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baab1c5f96484c18a9aebb43ab1f520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:59.447364  1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baab1c5f96484c18a9aebb43ab1f520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:00:59.452235  2184 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: Bootstrap starting.
I20240430 20:00:59.453245  2186 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c: Bootstrap starting.
I20240430 20:00:59.454610  2184 tablet_bootstrap.cc:654] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:59.455245  2186 tablet_bootstrap.cc:654] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:59.455353  2184 log.cc:824] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:59.455897  2186 log.cc:824] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:59.457000  2184 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: No bootstrap required, opened a new log
I20240430 20:00:59.457158  2184 ts_tablet_manager.cc:1367] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240430 20:00:59.457337  2186 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c: No bootstrap required, opened a new log
I20240430 20:00:59.457484  2186 ts_tablet_manager.cc:1367] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240430 20:00:59.460245  2184 raft_consensus.cc:348] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.460431  2184 raft_consensus.cc:374] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:59.460467  2184 raft_consensus.cc:729] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e563869b282a4389abf74929d93a015f, State: Initialized, Role: FOLLOWER
I20240430 20:00:59.460391  2186 raft_consensus.cc:348] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.460593  2186 raft_consensus.cc:374] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:59.460784  2186 raft_consensus.cc:729] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ba6f48197d4bf09cdc2af16980c15c, State: Initialized, Role: FOLLOWER
I20240430 20:00:59.460919  2184 consensus_queue.cc:260] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.461171  2186 consensus_queue.cc:260] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.461585  2036 heartbeater.cc:503] Master 127.0.197.190:45189 was elected leader, sending a full tablet report...
I20240430 20:00:59.461786  2184 ts_tablet_manager.cc:1398] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240430 20:00:59.461997  2186 ts_tablet_manager.cc:1398] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20240430 20:00:59.462049  2185 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Bootstrap starting.
I20240430 20:00:59.462234  2167 heartbeater.cc:503] Master 127.0.197.190:45189 was elected leader, sending a full tablet report...
I20240430 20:00:59.463876  2190 raft_consensus.cc:482] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:00:59.464013  2190 raft_consensus.cc:504] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.464468  2185 tablet_bootstrap.cc:654] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Neither blocks nor log segments found. Creating new log.
I20240430 20:00:59.465345  2185 log.cc:824] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Log is configured to *not* fsync() on all Append() calls
I20240430 20:00:59.466265  2190 leader_election.cc:288] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663), e563869b282a4389abf74929d93a015f (127.0.197.130:41937)
I20240430 20:00:59.467300  2185 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: No bootstrap required, opened a new log
I20240430 20:00:59.467507  2185 ts_tablet_manager.cc:1367] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
W20240430 20:00:59.469612  2037 tablet.cc:2369] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:00:59.469301  1859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab1c5f96484c18a9aebb43ab1f520d" candidate_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416de875ac014d1fbd60b90f51f626d2" is_pre_election: true
W20240430 20:00:59.470420  2055 leader_election.cc:341] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:00:59.471457  2185 raft_consensus.cc:348] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.471654  2185 raft_consensus.cc:374] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:00:59.471716  2185 raft_consensus.cc:729] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416de875ac014d1fbd60b90f51f626d2, State: Initialized, Role: FOLLOWER
I20240430 20:00:59.471702  1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab1c5f96484c18a9aebb43ab1f520d" candidate_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e563869b282a4389abf74929d93a015f" is_pre_election: true
I20240430 20:00:59.471956  1990 raft_consensus.cc:2454] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ba6f48197d4bf09cdc2af16980c15c in term 0.
I20240430 20:00:59.472152  2185 consensus_queue.cc:260] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.472324  2056 leader_election.cc:302] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e563869b282a4389abf74929d93a015f, f3ba6f48197d4bf09cdc2af16980c15c; no voters: 416de875ac014d1fbd60b90f51f626d2
I20240430 20:00:59.472541  2190 raft_consensus.cc:2781] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:00:59.472671  2190 raft_consensus.cc:482] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:00:59.472735  2190 raft_consensus.cc:3037] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:59.472774  1905 heartbeater.cc:503] Master 127.0.197.190:45189 was elected leader, sending a full tablet report...
I20240430 20:00:59.473033  2185 ts_tablet_manager.cc:1398] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240430 20:00:59.473620  2190 raft_consensus.cc:504] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.474013  2190 leader_election.cc:288] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [CANDIDATE]: Term 1 election: Requested vote from peers 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663), e563869b282a4389abf74929d93a015f (127.0.197.130:41937)
I20240430 20:00:59.474261  1859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab1c5f96484c18a9aebb43ab1f520d" candidate_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416de875ac014d1fbd60b90f51f626d2"
I20240430 20:00:59.474385  1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab1c5f96484c18a9aebb43ab1f520d" candidate_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e563869b282a4389abf74929d93a015f"
I20240430 20:00:59.474399  1859 raft_consensus.cc:3037] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:59.474503  1990 raft_consensus.cc:3037] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:00:59.475492  1990 raft_consensus.cc:2454] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ba6f48197d4bf09cdc2af16980c15c in term 1.
I20240430 20:00:59.475514  1859 raft_consensus.cc:2454] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ba6f48197d4bf09cdc2af16980c15c in term 1.
I20240430 20:00:59.475790  2056 leader_election.cc:302] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e563869b282a4389abf74929d93a015f, f3ba6f48197d4bf09cdc2af16980c15c; no voters: 
I20240430 20:00:59.476016  2190 raft_consensus.cc:2781] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:00:59.476253  2190 raft_consensus.cc:686] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [term 1 LEADER]: Becoming Leader. State: Replica: f3ba6f48197d4bf09cdc2af16980c15c, State: Running, Role: LEADER
I20240430 20:00:59.476545  2190 consensus_queue.cc:237] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:00:59.479401  1716 catalog_manager.cc:5484] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c reported cstate change: term changed from 0 to 1, leader changed from <none> to f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131). New cstate: current_term: 1 leader_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240430 20:00:59.487905   790 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:00:59.489228   790 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 416de875ac014d1fbd60b90f51f626d2 to finish bootstrapping
I20240430 20:00:59.494158   790 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e563869b282a4389abf74929d93a015f to finish bootstrapping
I20240430 20:00:59.498756   790 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f3ba6f48197d4bf09cdc2af16980c15c to finish bootstrapping
I20240430 20:00:59.503537   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 1777
I20240430 20:00:59.511479   790 linked_list-test-util.h:401] Running for: 30.000s
I20240430 20:00:59.511540   790 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 20:00:59.511559   790 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
W20240430 20:00:59.524556  2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111)
I20240430 20:00:59.524909  1990 raft_consensus.cc:1261] T baab1c5f96484c18a9aebb43ab1f520d P e563869b282a4389abf74929d93a015f [term 1 FOLLOWER]: Refusing update from remote peer f3ba6f48197d4bf09cdc2af16980c15c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240430 20:00:59.525420  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 20:00:59.525516  2194 consensus_queue.cc:1035] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:00:59.531258  2201 mvcc.cc:205] Tried to move back new op lower bound from 7022621735007350784 to 7022621734818824192. Current Snapshot: MvccSnapshot[applied={T|T < 7022621735007350784}]
I20240430 20:00:59.531726  2199 mvcc.cc:205] Tried to move back new op lower bound from 7022621735007350784 to 7022621734818824192. Current Snapshot: MvccSnapshot[applied={T|T < 7022621735007350784}]
W20240430 20:01:01.929764  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:01:02.719482  2190 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 12 ops behind the committed index 
W20240430 20:01:04.351816  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 20:01:04.770882  2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240430 20:01:06.175865  2194 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 566 ops behind the committed index  [suppressed 6 similar messages]
W20240430 20:01:06.686192  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 20:01:09.513435   790 linked_list-test-util.h:417] Sample at HT timestamp: 7022621775914409984 Inserted count: 79400
I20240430 20:01:09.567241  2194 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 1122 ops behind the committed index  [suppressed 5 similar messages]
W20240430 20:01:09.572718  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240430 20:01:10.118160  2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240430 20:01:11.986347  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 20:01:13.070335  2194 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 1689 ops behind the committed index  [suppressed 6 similar messages]
W20240430 20:01:14.448433  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 20:01:15.452185  2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240430 20:01:16.586539  2194 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 2263 ops behind the committed index  [suppressed 6 similar messages]
W20240430 20:01:17.025622  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 20:01:19.512010   790 linked_list-test-util.h:417] Sample at HT timestamp: 7022621816868249600 Inserted count: 159000
W20240430 20:01:19.555446  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240430 20:01:20.115281  2329 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 2809 ops behind the committed index  [suppressed 6 similar messages]
W20240430 20:01:20.680878  2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240430 20:01:22.113255  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 20:01:23.335399  2336 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 3309 ops behind the committed index  [suppressed 6 similar messages]
W20240430 20:01:24.313877  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240430 20:01:25.791399  2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240430 20:01:26.375545  2329 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 3799 ops behind the committed index  [suppressed 5 similar messages]
W20240430 20:01:26.816515  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240430 20:01:29.451053  2329 consensus_queue.cc:786] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c [LEADER]: Peer 416de875ac014d1fbd60b90f51f626d2 is lagging by at least 4285 ops behind the committed index  [suppressed 5 similar messages]
W20240430 20:01:29.456550  2055 consensus_peers.cc:485] T baab1c5f96484c18a9aebb43ab1f520d P f3ba6f48197d4bf09cdc2af16980c15c -> Peer 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663): Couldn't send request to peer 416de875ac014d1fbd60b90f51f626d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.129:36663: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240430 20:01:29.512461   790 linked_list-test-util.h:417] Sample at HT timestamp: 7022621857830301696 Inserted count: 237550
I20240430 20:01:29.512557   790 linked_list-test-util.h:421] Finished inserting list. Added 237550 in chain
I20240430 20:01:29.512580   790 linked_list-test-util.h:422] Last entries inserted had keys:
I20240430 20:01:29.512595   790 linked_list-test-util.h:424] 0: 5762154250149167104
I20240430 20:01:29.512609   790 linked_list-test-util.h:424] 1: -4826142986763698175
I20240430 20:01:29.512624   790 linked_list-test-util.h:424] 2: 3032726062497988610
I20240430 20:01:29.512646   790 linked_list-test-util.h:424] 3: -7555289699438166013
I20240430 20:01:29.512676   790 linked_list-test-util.h:424] 4: 6065311387507687428
I20240430 20:01:29.512699   790 linked_list-test-util.h:424] 5: -4522704374428467195
I20240430 20:01:29.512722   790 linked_list-test-util.h:424] 6: 3336164674833219590
I20240430 20:01:29.512745   790 linked_list-test-util.h:424] 7: -7251851087102935033
I20240430 20:01:29.512765   790 linked_list-test-util.h:424] 8: 6368749999842918408
I20240430 20:01:29.512794   790 linked_list-test-util.h:424] 9: -4219265762093236215
I20240430 20:01:29.512816   790 linked_list-test-util.h:424] 10: 3639603287168450570
I20240430 20:01:29.512831   790 linked_list-test-util.h:424] 11: -6948693949744414709
I20240430 20:01:29.512854   790 linked_list-test-util.h:424] 12: 6672188612178149388
I20240430 20:01:29.512876   790 linked_list-test-util.h:424] 13: -3915827149758005235
I20240430 20:01:29.512890   790 linked_list-test-util.h:424] 14: 3943041899503681550
I20240430 20:01:29.512913   790 linked_list-test-util.h:424] 15: -6645255337409183729
I20240430 20:01:29.512928   790 linked_list-test-util.h:424] 16: 6975627224513380368
I20240430 20:01:29.512950   790 linked_list-test-util.h:424] 17: -3612388537422774255
I20240430 20:01:29.512972   790 linked_list-test-util.h:424] 18: 4246058299373912082
I20240430 20:01:29.512987   790 linked_list-test-util.h:424] 19: -6341816725073952749
I20240430 20:01:29.513010   790 linked_list-test-util.h:424] 20: 7279065836848611348
I20240430 20:01:29.513025   790 linked_list-test-util.h:424] 21: -3308949925087543275
I20240430 20:01:29.513047   790 linked_list-test-util.h:424] 22: 4549496911709143062
I20240430 20:01:29.513062   790 linked_list-test-util.h:424] 23: -6038378112738721769
I20240430 20:01:29.513084   790 linked_list-test-util.h:424] 24: 7582504449183842328
I20240430 20:01:29.513100   790 linked_list-test-util.h:424] 25: -3005792787729022951
I20240430 20:01:29.513121   790 linked_list-test-util.h:424] 26: 4852935524044374042
I20240430 20:01:29.513144   790 linked_list-test-util.h:424] 27: 27074012257386523
I20240430 20:01:29.513159   790 linked_list-test-util.h:424] 28: 7885943061519073308
I20240430 20:01:29.513180   790 linked_list-test-util.h:424] 29: -2702354175393791971
I20240430 20:01:29.513213   790 linked_list-test-util.h:424] 30: 5156374136379605022
I20240430 20:01:29.513236   790 linked_list-test-util.h:424] 31: 330512624592617503
I20240430 20:01:29.513257   790 linked_list-test-util.h:424] 32: 8188959461389303840
I20240430 20:01:29.513273   790 linked_list-test-util.h:424] 33: -2398915563058560991
I20240430 20:01:29.513296   790 linked_list-test-util.h:424] 34: 5459812748714836002
I20240430 20:01:29.513312   790 linked_list-test-util.h:424] 35: 633951236927848483
I20240430 20:01:29.513334   790 linked_list-test-util.h:424] 36: 8492398073724534820
I20240430 20:01:29.513357   790 linked_list-test-util.h:424] 37: -2095476950723330011
I20240430 20:01:29.513372   790 linked_list-test-util.h:424] 38: 5763251361050066982
I20240430 20:01:29.513396   790 linked_list-test-util.h:424] 39: 937108374286368807
I20240430 20:01:29.513411   790 linked_list-test-util.h:424] 40: 8795836686059765800
I20240430 20:01:29.513433   790 linked_list-test-util.h:424] 41: -1792038338388099031
I20240430 20:01:29.513456   790 linked_list-test-util.h:424] 42: 6066408498408587306
I20240430 20:01:29.513471   790 linked_list-test-util.h:424] 43: 1240546986621599787
I20240430 20:01:29.513495   790 linked_list-test-util.h:424] 44: 9099275298394996780
I20240430 20:01:29.513511   790 linked_list-test-util.h:424] 45: -1488599726052868051
I20240430 20:01:29.513533   790 linked_list-test-util.h:424] 46: 6369847110743818286
I20240430 20:01:29.513549   790 linked_list-test-util.h:424] 47: 1543985598956830767
I20240430 20:01:29.513572   790 linked_list-test-util.h:424] 48: -9044030162979323856
I20240430 20:01:29.513594   790 linked_list-test-util.h:424] 49: -1185442588694347727
I20240430 20:01:29.544850   790 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
/tmp/dist-test-taskfMcbc6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.129:36663
--local_ip_for_outbound_sockets=127.0.197.129
--tserver_master_addrs=127.0.197.190:45189
--webserver_port=37753
--webserver_interface=127.0.197.129
--builtin_ntp_servers=127.0.197.148:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 20:01:29.655880  2397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:01:29.656102  2397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:01:29.656150  2397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:01:29.659149  2397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:01:29.659329  2397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.129
I20240430 20:01:29.662801  2397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.148:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.129:36663
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.0.197.129
--webserver_port=37753
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.0.197.190:45189
--never_fsync=true
--heap_profile_path=/tmp/kudu.2397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.129
--log_dir=/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 19:43:14 UTC on 1a15035a6830
build id 61
I20240430 20:01:29.663725  2397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:01:29.664872  2397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:01:29.674093  2405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:01:29.674587  2402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:01:29.675552  2397 server_base.cc:1034] running on GCE node
W20240430 20:01:29.676323  2403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:01:29.676716  2397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:01:29.677332  2397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:01:29.678483  2397 hybrid_clock.cc:648] HybridClock initialized: now 1714507289678461 us; error 38 us; skew 500 ppm
I20240430 20:01:29.685317  2397 webserver.cc:460] Webserver started at http://127.0.197.129:37753/ using document root <none> and password file <none>
I20240430 20:01:29.685895  2397 fs_manager.cc:359] Metadata directory not provided
I20240430 20:01:29.685972  2397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:01:29.689378  2397 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.001s	sys 0.000s
I20240430 20:01:29.691469  2411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 20:01:29.692494  2397 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240430 20:01:29.692600  2397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
uuid: "416de875ac014d1fbd60b90f51f626d2"
format_stamp: "Formatted at 2024-04-30 20:00:59 on dist-test-slave-81l2"
I20240430 20:01:29.693014  2397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:01:29.709347  2397 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 20:01:29.709964  2397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:01:29.710121  2397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:01:29.710634  2397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:01:29.711755  2418 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240430 20:01:29.714759  2397 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240430 20:01:29.714823  2397 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s	user 0.000s	sys 0.000s
I20240430 20:01:29.714895  2397 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240430 20:01:29.716303  2397 ts_tablet_manager.cc:590] Registered 1 tablets
I20240430 20:01:29.716348  2397 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
I20240430 20:01:29.716565  2418 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Bootstrap starting.
I20240430 20:01:29.726667  2397 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.129:36663
I20240430 20:01:29.726697  2525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.129:36663 every 8 connection(s)
I20240430 20:01:29.727699  2397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/data/info.pb
I20240430 20:01:29.731472   790 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu as pid 2397
I20240430 20:01:29.731693  2418 log.cc:824] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Log is configured to *not* fsync() on all Append() calls
I20240430 20:01:29.733186  2418 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 20:01:29.733363  2418 log.cc:1225] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskfMcbc6/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714507222116962-790-0/linked-list-cluster/ts-0/wal/wals/baab1c5f96484c18a9aebb43ab1f520d.recovery-1714507289733312
I20240430 20:01:29.733409  2418 tablet_bootstrap.cc:492] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Bootstrap complete.
I20240430 20:01:29.733785  2418 ts_tablet_manager.cc:1367] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.010s
I20240430 20:01:29.737411  2526 heartbeater.cc:347] Connected to a master server at 127.0.197.190:45189
I20240430 20:01:29.737636  2526 heartbeater.cc:464] Registering TS with master...
I20240430 20:01:29.737322  2418 raft_consensus.cc:348] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:01:29.737707  2418 raft_consensus.cc:729] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416de875ac014d1fbd60b90f51f626d2, State: Initialized, Role: FOLLOWER
I20240430 20:01:29.738168  2418 consensus_queue.cc:260] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:01:29.738207  2526 heartbeater.cc:511] Master 127.0.197.190:45189 requested a full tablet report, sending...
I20240430 20:01:29.738979  2418 ts_tablet_manager.cc:1398] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240430 20:01:29.740160  1711 ts_manager.cc:194] Re-registered known tserver with Master: 416de875ac014d1fbd60b90f51f626d2 (127.0.197.129:36663)
I20240430 20:01:29.741129  1711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.129:54507
I20240430 20:01:29.742856  2526 heartbeater.cc:503] Master 127.0.197.190:45189 was elected leader, sending a full tablet report...
I20240430 20:01:29.748960   790 cluster_itest_util.cc:256] Not converged past 4751 yet: 1.4782 1.4782 0.0
I20240430 20:01:29.851732   790 cluster_itest_util.cc:256] Not converged past 4751 yet: 1.4782 1.4782 0.0
I20240430 20:01:30.104609   790 cluster_itest_util.cc:256] Not converged past 4751 yet: 1.4782 1.4782 1.976
I20240430 20:01:30.407188   790 cluster_itest_util.cc:256] Not converged past 4751 yet: 1.4782 1.4782 1.1984
I20240430 20:01:30.428472  2536 mvcc.cc:205] Tried to move back new op lower bound from 7022621747767984128 to 7022621734818824192. Current Snapshot: MvccSnapshot[applied={T|T < 7022621747767984128}]
I20240430 20:01:30.809592   790 cluster_itest_util.cc:256] Not converged past 4751 yet: 1.4782 1.4782 1.3458
I20240430 20:01:31.312670   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 1908
I20240430 20:01:31.324723   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 2039
I20240430 20:01:31.339742   790 linked_list-test-util.h:509] Verifying Snapshot: P: 1714507269510354 usec, L: 1 Expected Rows: 79400
I20240430 20:01:31.351598   790 meta_cache.cc:1510] marking tablet server f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) as failed
W20240430 20:01:31.351706   790 meta_cache.cc:302] tablet baab1c5f96484c18a9aebb43ab1f520d: replica f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
I20240430 20:01:31.360005   790 meta_cache.cc:1510] marking tablet server e563869b282a4389abf74929d93a015f (127.0.197.130:41937) as failed
I20240430 20:01:31.516563   790 linked_list-test-util.h:528] Calling callback on tserver 416de875ac014d1fbd60b90f51f626d2
I20240430 20:01:32.576174  2546 raft_consensus.cc:482] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3ba6f48197d4bf09cdc2af16980c15c)
I20240430 20:01:32.577765  2546 raft_consensus.cc:504] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:01:32.589844  2546 leader_election.cc:288] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131), e563869b282a4389abf74929d93a015f (127.0.197.130:41937)
W20240430 20:01:32.590305  2414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
W20240430 20:01:32.591430  2414 leader_election.cc:334] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131): Network error: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
W20240430 20:01:32.592063  2414 leader_election.cc:334] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e563869b282a4389abf74929d93a015f (127.0.197.130:41937): Network error: Client connection negotiation failed: client connection to 127.0.197.130:41937: connect: Connection refused (error 111)
I20240430 20:01:32.592136  2414 leader_election.cc:302] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 416de875ac014d1fbd60b90f51f626d2; no voters: e563869b282a4389abf74929d93a015f, f3ba6f48197d4bf09cdc2af16980c15c
I20240430 20:01:32.592403  2546 raft_consensus.cc:2726] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:01:33.034124   790 linked_list-test-util.h:767] Done collecting results (79400 rows in 1517.66ms)
I20240430 20:01:33.117653   790 linked_list-test-util.h:556] Snapshot: P: 1714507269510354 usec, L: 1 verified. Result: OK
I20240430 20:01:33.121042   790 linked_list-test-util.h:509] Verifying Snapshot: P: 1714507279508850 usec, L: 1 Expected Rows: 159000
I20240430 20:01:33.128803   790 meta_cache.cc:1510] marking tablet server f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) as failed
W20240430 20:01:33.128921   790 meta_cache.cc:302] tablet baab1c5f96484c18a9aebb43ab1f520d: replica f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
I20240430 20:01:33.131932   790 meta_cache.cc:1510] marking tablet server e563869b282a4389abf74929d93a015f (127.0.197.130:41937) as failed
I20240430 20:01:33.214773   790 linked_list-test-util.h:528] Calling callback on tserver 416de875ac014d1fbd60b90f51f626d2
I20240430 20:01:34.138715  2551 raft_consensus.cc:482] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3ba6f48197d4bf09cdc2af16980c15c)
I20240430 20:01:34.138854  2551 raft_consensus.cc:504] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:01:34.139981  2551 leader_election.cc:288] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131), e563869b282a4389abf74929d93a015f (127.0.197.130:41937)
W20240430 20:01:34.159516  2414 leader_election.cc:334] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131): Network error: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
W20240430 20:01:34.159989  2414 leader_election.cc:334] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e563869b282a4389abf74929d93a015f (127.0.197.130:41937): Network error: Client connection negotiation failed: client connection to 127.0.197.130:41937: connect: Connection refused (error 111)
I20240430 20:01:34.160065  2414 leader_election.cc:302] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 416de875ac014d1fbd60b90f51f626d2; no voters: e563869b282a4389abf74929d93a015f, f3ba6f48197d4bf09cdc2af16980c15c
I20240430 20:01:34.160553  2551 raft_consensus.cc:2726] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:01:35.554941   790 linked_list-test-util.h:767] Done collecting results (159000 rows in 2340.25ms)
I20240430 20:01:35.673827   790 linked_list-test-util.h:556] Snapshot: P: 1714507279508850 usec, L: 1 verified. Result: OK
I20240430 20:01:35.675931   790 linked_list-test-util.h:509] Verifying Snapshot: P: 1714507289509351 usec, L: 1 Expected Rows: 237550
I20240430 20:01:35.676854   790 meta_cache.cc:1510] marking tablet server f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) as failed
W20240430 20:01:35.676923   790 meta_cache.cc:302] tablet baab1c5f96484c18a9aebb43ab1f520d: replica f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
I20240430 20:01:35.678566   790 meta_cache.cc:1510] marking tablet server e563869b282a4389abf74929d93a015f (127.0.197.130:41937) as failed
I20240430 20:01:35.683538   790 linked_list-test-util.h:528] Calling callback on tserver 416de875ac014d1fbd60b90f51f626d2
I20240430 20:01:37.228479  2556 raft_consensus.cc:482] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3ba6f48197d4bf09cdc2af16980c15c)
I20240430 20:01:37.228615  2556 raft_consensus.cc:504] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416de875ac014d1fbd60b90f51f626d2" member_type: VOTER last_known_addr { host: "127.0.197.129" port: 36663 } } peers { permanent_uuid: "f3ba6f48197d4bf09cdc2af16980c15c" member_type: VOTER last_known_addr { host: "127.0.197.131" port: 34131 } } peers { permanent_uuid: "e563869b282a4389abf74929d93a015f" member_type: VOTER last_known_addr { host: "127.0.197.130" port: 41937 } }
I20240430 20:01:37.228972  2556 leader_election.cc:288] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131), e563869b282a4389abf74929d93a015f (127.0.197.130:41937)
W20240430 20:01:37.230120  2414 leader_election.cc:334] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131): Network error: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
W20240430 20:01:37.230234  2414 leader_election.cc:334] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e563869b282a4389abf74929d93a015f (127.0.197.130:41937): Network error: Client connection negotiation failed: client connection to 127.0.197.130:41937: connect: Connection refused (error 111)
I20240430 20:01:37.230281  2414 leader_election.cc:302] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 416de875ac014d1fbd60b90f51f626d2; no voters: e563869b282a4389abf74929d93a015f, f3ba6f48197d4bf09cdc2af16980c15c
I20240430 20:01:37.230405  2556 raft_consensus.cc:2726] T baab1c5f96484c18a9aebb43ab1f520d P 416de875ac014d1fbd60b90f51f626d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:01:37.774233   790 linked_list-test-util.h:767] Done collecting results (237550 rows in 2090.77ms)
I20240430 20:01:37.954640   790 linked_list-test-util.h:556] Snapshot: P: 1714507289509351 usec, L: 1 verified. Result: OK
I20240430 20:01:37.956668   790 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 237550
I20240430 20:01:37.957603   790 meta_cache.cc:1510] marking tablet server f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) as failed
W20240430 20:01:37.957667   790 meta_cache.cc:302] tablet baab1c5f96484c18a9aebb43ab1f520d: replica f3ba6f48197d4bf09cdc2af16980c15c (127.0.197.131:34131) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.0.197.131:34131: connect: Connection refused (error 111)
I20240430 20:01:37.959252   790 meta_cache.cc:1510] marking tablet server e563869b282a4389abf74929d93a015f (127.0.197.130:41937) as failed
I20240430 20:01:37.960839   790 linked_list-test-util.h:528] Calling callback on tserver 416de875ac014d1fbd60b90f51f626d2
I20240430 20:01:40.043816   790 linked_list-test-util.h:767] Done collecting results (237550 rows in 2083.05ms)
I20240430 20:01:40.224737   790 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240430 20:01:40.225045   790 linked_list-test-util.h:692] Successfully verified 237550 rows
I20240430 20:01:40.225615   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 2397
W20240430 20:01:40.225615  2171 client.cc:2025] Couldn't close scanner 5994b9c854294634a766cc83bf808dcd: Service unavailable: reactor is shutting down (error 108)
I20240430 20:01:40.236464   790 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfMcbc6/build/debug/bin/kudu with pid 1686
2024-04-30T20:01:40Z chronyd exiting
[       OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (41637 ms)
[----------] 2 tests from LinkedListTest (78133 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (78133 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LinkedListTest.TestLoadAndVerify

 1 FAILED TEST
I20240430 20:01:40.262704   790 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 1 messages since previous log ~2 seconds ago