LinkedListTest.TestLoadAndVerify: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: f2aa0f551289403caa641628dd390846, Rpc address: host: "127.23.217.66" port: 45035 (Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111))
I20240430 10:33:40.658015 24421 webui_checker.cc:88] shutting down CURL thread
W20240430 10:33:40.797094 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 10:33:40.887014 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 10:33:41.267515 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240430 10:33:41.375916 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:41.776963 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:41.784718 25317 consensus_queue.cc:786] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 26 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:41.789422 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 24524
I20240430 10:33:41.799050 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 24786
I20240430 10:33:41.808035 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 24433
2024-04-30T10:33:41Z chronyd exiting
I20240430 10:33:41.825536 24421 test_util.cc:175] -----------------------------------------------
I20240430 10:33:41.825588 24421 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN      ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 10:33:04.577776 24421 test_util.cc:268] Using random seed: -450574113
I20240430 10:33:04.584537 24421 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 10:33:04.584589 24421 linked_list-test.cc:101] --------------
I20240430 10:33:04.584605 24421 linked_list-test.cc:102] 50 chains
I20240430 10:33:04.584642 24421 linked_list-test.cc:103] 3 tablets
I20240430 10:33:04.584656 24421 linked_list-test.cc:104] Mutations on
I20240430 10:33:04.584676 24421 linked_list-test.cc:105] --------------
I20240430 10:33:04.584756 24421 ts_itest-base.cc:112] Starting cluster with:
I20240430 10:33:04.584828 24421 ts_itest-base.cc:113] --------------
I20240430 10:33:04.584857 24421 ts_itest-base.cc:114] 3 tablet servers
I20240430 10:33:04.584879 24421 ts_itest-base.cc:115] 3 replicas per TS
I20240430 10:33:04.584903 24421 ts_itest-base.cc:116] --------------
2024-04-30T10:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T10:33:04Z Disabled control of system clock
I20240430 10:33:04.594789 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.217.126:44421
--webserver_interface=127.23.217.126
--webserver_port=0
--builtin_ntp_servers=127.23.217.84:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.217.126:44421
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 10:33:04.698822 24433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:04.699072 24433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:04.699118 24433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:04.702023 24433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 10:33:04.702081 24433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:04.702121 24433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 10:33:04.702137 24433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 10:33:04.705667 24433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.217.126:44421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.217.126:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.23.217.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.24433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:04.706542 24433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:04.707993 24433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:04.715704 24441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:04.716470 24433 server_base.cc:1034] running on GCE node
W20240430 10:33:04.716931 24438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:04.717541 24439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:04.718037 24433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:04.719018 24433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:04.720192 24433 hybrid_clock.cc:648] HybridClock initialized: now 1714473184720180 us; error 29 us; skew 500 ppm
I20240430 10:33:04.725850 24433 webserver.cc:460] Webserver started at http://127.23.217.126:44427/ using document root <none> and password file <none>
I20240430 10:33:04.726382 24433 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:04.726435 24433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:04.726603 24433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:04.728273 24433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/instance:
uuid: "960358990c09452681d18db6ef0dcf24"
format_stamp: "Formatted at 2024-04-30 10:33:04 on dist-test-slave-k5tk"
I20240430 10:33:04.728734 24433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal/instance:
uuid: "960358990c09452681d18db6ef0dcf24"
format_stamp: "Formatted at 2024-04-30 10:33:04 on dist-test-slave-k5tk"
I20240430 10:33:04.731933 24433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.002s
I20240430 10:33:04.733976 24447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:04.735005 24433 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240430 10:33:04.735105 24433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
uuid: "960358990c09452681d18db6ef0dcf24"
format_stamp: "Formatted at 2024-04-30 10:33:04 on dist-test-slave-k5tk"
I20240430 10:33:04.735175 24433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:04.752228 24433 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:04.752830 24433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:04.753000 24433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:04.759492 24433 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.126:44421
I20240430 10:33:04.759496 24499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.126:44421 every 8 connection(s)
I20240430 10:33:04.760397 24433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/info.pb
I20240430 10:33:04.762980 24500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:04.767868 24500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24: Bootstrap starting.
I20240430 10:33:04.770221 24500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:04.770499 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 24433
I20240430 10:33:04.770648 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal/instance
I20240430 10:33:04.771139 24500 log.cc:824] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:04.773113 24500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24: No bootstrap required, opened a new log
I20240430 10:33:04.776419 24500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960358990c09452681d18db6ef0dcf24" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 44421 } }
I20240430 10:33:04.776607 24500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:04.776674 24500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960358990c09452681d18db6ef0dcf24, State: Initialized, Role: FOLLOWER
I20240430 10:33:04.777107 24500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960358990c09452681d18db6ef0dcf24" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 44421 } }
I20240430 10:33:04.777228 24500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 10:33:04.777292 24500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 10:33:04.777391 24500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:04.778316 24500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960358990c09452681d18db6ef0dcf24" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 44421 } }
I20240430 10:33:04.778640 24500 leader_election.cc:302] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 960358990c09452681d18db6ef0dcf24; no voters: 
I20240430 10:33:04.778971 24500 leader_election.cc:288] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240430 10:33:04.779062 24505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:33:04.779289 24505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [term 1 LEADER]: Becoming Leader. State: Replica: 960358990c09452681d18db6ef0dcf24, State: Running, Role: LEADER
I20240430 10:33:04.779637 24505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960358990c09452681d18db6ef0dcf24" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 44421 } }
I20240430 10:33:04.780097 24500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 10:33:04.781112 24507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 960358990c09452681d18db6ef0dcf24. Latest consensus state: current_term: 1 leader_uuid: "960358990c09452681d18db6ef0dcf24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960358990c09452681d18db6ef0dcf24" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 44421 } } }
I20240430 10:33:04.781112 24506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "960358990c09452681d18db6ef0dcf24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960358990c09452681d18db6ef0dcf24" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 44421 } } }
I20240430 10:33:04.781248 24506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [sys.catalog]: This master's current role is: LEADER
I20240430 10:33:04.781657 24514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 10:33:04.781248 24507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24 [sys.catalog]: This master's current role is: LEADER
I20240430 10:33:04.784812 24514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 10:33:04.789786 24514 catalog_manager.cc:1347] Generated new cluster ID: 47122a2789f04ec4aa6d3b622a82beb6
I20240430 10:33:04.789850 24514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 10:33:04.813020 24514 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 10:33:04.813921 24514 catalog_manager.cc:1504] Loading token signing keys...
I20240430 10:33:04.819012 24514 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 960358990c09452681d18db6ef0dcf24: Generated new TSK 0
I20240430 10:33:04.819679 24514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 10:33:04.823611 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.65:0
--local_ip_for_outbound_sockets=127.23.217.65
--webserver_interface=127.23.217.65
--webserver_port=0
--tserver_master_addrs=127.23.217.126:44421
--builtin_ntp_servers=127.23.217.84:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240430 10:33:04.927495 24524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:04.927731 24524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:04.927798 24524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:04.930779 24524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:04.930850 24524 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 10:33:04.930866 24524 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 10:33:04.930886 24524 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 10:33:04.930982 24524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.65
I20240430 10:33:04.934504 24524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.23.217.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.23.217.126:44421
--never_fsync=true
--heap_profile_path=/tmp/kudu.24524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.65
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:04.935369 24524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:04.936439 24524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:04.944873 24529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:04.945439 24530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:04.946053 24532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:04.946676 24524 server_base.cc:1034] running on GCE node
I20240430 10:33:04.947088 24524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:04.947645 24524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:04.948808 24524 hybrid_clock.cc:648] HybridClock initialized: now 1714473184948799 us; error 23 us; skew 500 ppm
I20240430 10:33:04.950637 24524 webserver.cc:460] Webserver started at http://127.23.217.65:36497/ using document root <none> and password file <none>
I20240430 10:33:04.951196 24524 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:04.951253 24524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:04.951402 24524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:04.952976 24524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/instance:
uuid: "cf979890494e40c8b3207effd7c83cef"
format_stamp: "Formatted at 2024-04-30 10:33:04 on dist-test-slave-k5tk"
I20240430 10:33:04.953414 24524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal/instance:
uuid: "cf979890494e40c8b3207effd7c83cef"
format_stamp: "Formatted at 2024-04-30 10:33:04 on dist-test-slave-k5tk"
I20240430 10:33:04.956579 24524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240430 10:33:04.958560 24538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:04.959569 24524 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:33:04.959661 24524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
uuid: "cf979890494e40c8b3207effd7c83cef"
format_stamp: "Formatted at 2024-04-30 10:33:04 on dist-test-slave-k5tk"
I20240430 10:33:04.959733 24524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:04.984972 24524 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:04.985584 24524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:04.985716 24524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:04.986200 24524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:33:04.987074 24524 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:33:04.987130 24524 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:04.987177 24524 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:33:04.987202 24524 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:04.995569 24524 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.65:34067
I20240430 10:33:04.995604 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.65:34067 every 8 connection(s)
I20240430 10:33:04.996776 24524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
I20240430 10:33:04.999269 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 24524
I20240430 10:33:04.999384 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal/instance
I20240430 10:33:05.001454 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.66:0
--local_ip_for_outbound_sockets=127.23.217.66
--webserver_interface=127.23.217.66
--webserver_port=0
--tserver_master_addrs=127.23.217.126:44421
--builtin_ntp_servers=127.23.217.84:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 10:33:05.005761 24652 heartbeater.cc:347] Connected to a master server at 127.23.217.126:44421
I20240430 10:33:05.005972 24652 heartbeater.cc:464] Registering TS with master...
I20240430 10:33:05.006477 24652 heartbeater.cc:511] Master 127.23.217.126:44421 requested a full tablet report, sending...
I20240430 10:33:05.008081 24464 ts_manager.cc:194] Registered new tserver with Master: cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067)
I20240430 10:33:05.009838 24464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.65:33671
W20240430 10:33:05.107352 24655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:05.107558 24655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:05.107611 24655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:05.110498 24655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:05.110569 24655 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 10:33:05.110587 24655 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 10:33:05.110610 24655 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 10:33:05.110649 24655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.66
I20240430 10:33:05.114127 24655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.23.217.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.23.217.126:44421
--never_fsync=true
--heap_profile_path=/tmp/kudu.24655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.66
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:05.114941 24655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:05.115980 24655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:05.124140 24663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:05.124776 24660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:05.125376 24661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:05.126072 24655 server_base.cc:1034] running on GCE node
I20240430 10:33:05.126499 24655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:05.127110 24655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:05.128280 24655 hybrid_clock.cc:648] HybridClock initialized: now 1714473185128270 us; error 27 us; skew 500 ppm
I20240430 10:33:05.130156 24655 webserver.cc:460] Webserver started at http://127.23.217.66:42135/ using document root <none> and password file <none>
I20240430 10:33:05.130700 24655 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:05.130766 24655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:05.131007 24655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:05.132706 24655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/instance:
uuid: "f2aa0f551289403caa641628dd390846"
format_stamp: "Formatted at 2024-04-30 10:33:05 on dist-test-slave-k5tk"
I20240430 10:33:05.133196 24655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal/instance:
uuid: "f2aa0f551289403caa641628dd390846"
format_stamp: "Formatted at 2024-04-30 10:33:05 on dist-test-slave-k5tk"
I20240430 10:33:05.136466 24655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240430 10:33:05.138530 24669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:05.139518 24655 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:33:05.139626 24655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
uuid: "f2aa0f551289403caa641628dd390846"
format_stamp: "Formatted at 2024-04-30 10:33:05 on dist-test-slave-k5tk"
I20240430 10:33:05.139703 24655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:05.171056 24655 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:05.171747 24655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:05.171929 24655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:05.172611 24655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:33:05.173568 24655 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:33:05.173621 24655 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:05.173679 24655 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:33:05.173738 24655 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:05.183509 24655 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.66:45035
I20240430 10:33:05.183562 24782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.66:45035 every 8 connection(s)
I20240430 10:33:05.184418 24655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/info.pb
I20240430 10:33:05.187464 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 24655
I20240430 10:33:05.187567 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal/instance
I20240430 10:33:05.189627 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.67:0
--local_ip_for_outbound_sockets=127.23.217.67
--webserver_interface=127.23.217.67
--webserver_port=0
--tserver_master_addrs=127.23.217.126:44421
--builtin_ntp_servers=127.23.217.84:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240430 10:33:05.193500 24783 heartbeater.cc:347] Connected to a master server at 127.23.217.126:44421
I20240430 10:33:05.193723 24783 heartbeater.cc:464] Registering TS with master...
I20240430 10:33:05.194231 24783 heartbeater.cc:511] Master 127.23.217.126:44421 requested a full tablet report, sending...
I20240430 10:33:05.195269 24464 ts_manager.cc:194] Registered new tserver with Master: f2aa0f551289403caa641628dd390846 (127.23.217.66:45035)
I20240430 10:33:05.196125 24464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.66:47655
W20240430 10:33:05.295181 24786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:05.295404 24786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:05.295464 24786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:05.298455 24786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:05.298538 24786 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240430 10:33:05.298561 24786 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240430 10:33:05.298580 24786 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240430 10:33:05.298621 24786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.67
I20240430 10:33:05.302193 24786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.23.217.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.23.217.126:44421
--never_fsync=true
--heap_profile_path=/tmp/kudu.24786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.67
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:05.303110 24786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:05.304195 24786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:05.312456 24792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:05.313123 24794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:05.313664 24791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:05.314381 24786 server_base.cc:1034] running on GCE node
I20240430 10:33:05.314729 24786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:05.315285 24786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:05.316439 24786 hybrid_clock.cc:648] HybridClock initialized: now 1714473185316426 us; error 27 us; skew 500 ppm
I20240430 10:33:05.318212 24786 webserver.cc:460] Webserver started at http://127.23.217.67:44449/ using document root <none> and password file <none>
I20240430 10:33:05.318809 24786 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:05.318914 24786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:05.319114 24786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:05.320753 24786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/instance:
uuid: "2dffd8c702cd4fdba865ead4911f8584"
format_stamp: "Formatted at 2024-04-30 10:33:05 on dist-test-slave-k5tk"
I20240430 10:33:05.321211 24786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal/instance:
uuid: "2dffd8c702cd4fdba865ead4911f8584"
format_stamp: "Formatted at 2024-04-30 10:33:05 on dist-test-slave-k5tk"
I20240430 10:33:05.324343 24786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.003s
I20240430 10:33:05.326347 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:05.327368 24786 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:33:05.327468 24786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
uuid: "2dffd8c702cd4fdba865ead4911f8584"
format_stamp: "Formatted at 2024-04-30 10:33:05 on dist-test-slave-k5tk"
I20240430 10:33:05.327553 24786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:05.337280 24786 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:05.337842 24786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:05.337994 24786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:05.338456 24786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:33:05.339349 24786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:33:05.339394 24786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:05.339429 24786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:33:05.339459 24786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:05.347806 24786 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.67:41777
I20240430 10:33:05.347827 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.67:41777 every 8 connection(s)
I20240430 10:33:05.348686 24786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/info.pb
I20240430 10:33:05.355031 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 24786
I20240430 10:33:05.355135 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal/instance
I20240430 10:33:05.356873 24914 heartbeater.cc:347] Connected to a master server at 127.23.217.126:44421
I20240430 10:33:05.357084 24914 heartbeater.cc:464] Registering TS with master...
I20240430 10:33:05.357626 24914 heartbeater.cc:511] Master 127.23.217.126:44421 requested a full tablet report, sending...
I20240430 10:33:05.358513 24464 ts_manager.cc:194] Registered new tserver with Master: 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:05.359082 24464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.67:56859
I20240430 10:33:05.369318 24421 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 10:33:05.380362 24464 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40000:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240430 10:33:05.383327 24464 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 10:33:05.399827 24584 tablet_service.cc:1467] Processing CreateTablet for tablet e1b2f890834447e1b34a35e5bcf1203b (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 10:33:05.399827 24846 tablet_service.cc:1467] Processing CreateTablet for tablet e1b2f890834447e1b34a35e5bcf1203b (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 10:33:05.400317 24716 tablet_service.cc:1467] Processing CreateTablet for tablet 07177c6fdc5742458352ef97b02bebb9 (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 10:33:05.401013 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2f890834447e1b34a35e5bcf1203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.401013 24584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2f890834447e1b34a35e5bcf1203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.401194 24716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07177c6fdc5742458352ef97b02bebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.403201 24717 tablet_service.cc:1467] Processing CreateTablet for tablet 6ebcf145701c453694b3c0e260fd396b (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 10:33:05.403297 24702 tablet_service.cc:1467] Processing CreateTablet for tablet e1b2f890834447e1b34a35e5bcf1203b (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240430 10:33:05.403465 24717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ebcf145701c453694b3c0e260fd396b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.403892 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2f890834447e1b34a35e5bcf1203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.404763 24585 tablet_service.cc:1467] Processing CreateTablet for tablet 6ebcf145701c453694b3c0e260fd396b (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 10:33:05.404695 24847 tablet_service.cc:1467] Processing CreateTablet for tablet 6ebcf145701c453694b3c0e260fd396b (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240430 10:33:05.405512 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ebcf145701c453694b3c0e260fd396b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.399828 24586 tablet_service.cc:1467] Processing CreateTablet for tablet 07177c6fdc5742458352ef97b02bebb9 (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 10:33:05.407222 24586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07177c6fdc5742458352ef97b02bebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.407580 24585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ebcf145701c453694b3c0e260fd396b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.409014 24933 tablet_bootstrap.cc:492] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef: Bootstrap starting.
I20240430 10:33:05.409353 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 07177c6fdc5742458352ef97b02bebb9 (DEFAULT_TABLE table=TestTable [id=4eb396e7a4e74006aebd7f9f040acea5]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240430 10:33:05.409593 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07177c6fdc5742458352ef97b02bebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:05.410697 24931 tablet_bootstrap.cc:492] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584: Bootstrap starting.
I20240430 10:33:05.411383 24933 tablet_bootstrap.cc:654] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.411410 24932 tablet_bootstrap.cc:492] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846: Bootstrap starting.
I20240430 10:33:05.412525 24933 log.cc:824] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:05.412990 24931 tablet_bootstrap.cc:654] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.413059 24932 tablet_bootstrap.cc:654] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.413918 24932 log.cc:824] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:05.413975 24931 log.cc:824] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:05.414484 24933 tablet_bootstrap.cc:492] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef: No bootstrap required, opened a new log
I20240430 10:33:05.414708 24933 ts_tablet_manager.cc:1367] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240430 10:33:05.415426 24932 tablet_bootstrap.cc:492] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846: No bootstrap required, opened a new log
I20240430 10:33:05.415537 24931 tablet_bootstrap.cc:492] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584: No bootstrap required, opened a new log
I20240430 10:33:05.415592 24932 ts_tablet_manager.cc:1367] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240430 10:33:05.415702 24931 ts_tablet_manager.cc:1367] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240430 10:33:05.417821 24932 raft_consensus.cc:348] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.417994 24932 raft_consensus.cc:374] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.417886 24933 raft_consensus.cc:348] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.418047 24932 raft_consensus.cc:729] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aa0f551289403caa641628dd390846, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.418061 24933 raft_consensus.cc:374] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.418112 24933 raft_consensus.cc:729] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf979890494e40c8b3207effd7c83cef, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.418473 24932 consensus_queue.cc:260] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.418545 24933 consensus_queue.cc:260] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.418663 24931 raft_consensus.cc:348] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.418848 24931 raft_consensus.cc:374] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.418923 24931 raft_consensus.cc:729] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dffd8c702cd4fdba865ead4911f8584, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.419137 24783 heartbeater.cc:503] Master 127.23.217.126:44421 was elected leader, sending a full tablet report...
I20240430 10:33:05.419301 24931 consensus_queue.cc:260] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.419405 24932 ts_tablet_manager.cc:1398] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240430 10:33:05.419641 24932 tablet_bootstrap.cc:492] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846: Bootstrap starting.
I20240430 10:33:05.419838 24933 ts_tablet_manager.cc:1398] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.005s
I20240430 10:33:05.420056 24931 ts_tablet_manager.cc:1398] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20240430 10:33:05.420270 24652 heartbeater.cc:503] Master 127.23.217.126:44421 was elected leader, sending a full tablet report...
I20240430 10:33:05.420349 24931 tablet_bootstrap.cc:492] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584: Bootstrap starting.
I20240430 10:33:05.421196 24914 heartbeater.cc:503] Master 127.23.217.126:44421 was elected leader, sending a full tablet report...
I20240430 10:33:05.421278 24931 tablet_bootstrap.cc:654] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.422086 24933 tablet_bootstrap.cc:492] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef: Bootstrap starting.
I20240430 10:33:05.422399 24931 tablet_bootstrap.cc:492] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584: No bootstrap required, opened a new log
I20240430 10:33:05.422489 24931 ts_tablet_manager.cc:1367] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20240430 10:33:05.423125 24933 tablet_bootstrap.cc:654] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.423082 24931 raft_consensus.cc:348] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.423241 24931 raft_consensus.cc:374] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.423296 24931 raft_consensus.cc:729] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dffd8c702cd4fdba865ead4911f8584, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.423413 24931 consensus_queue.cc:260] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.423674 24931 ts_tablet_manager.cc:1398] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 10:33:05.423838 24931 tablet_bootstrap.cc:492] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584: Bootstrap starting.
I20240430 10:33:05.424669 24933 tablet_bootstrap.cc:492] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef: No bootstrap required, opened a new log
I20240430 10:33:05.424746 24933 ts_tablet_manager.cc:1367] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240430 10:33:05.424845 24931 tablet_bootstrap.cc:654] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.424906 24932 tablet_bootstrap.cc:654] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.425279 24933 raft_consensus.cc:348] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.425392 24933 raft_consensus.cc:374] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.425427 24933 raft_consensus.cc:729] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf979890494e40c8b3207effd7c83cef, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.425570 24933 consensus_queue.cc:260] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.425841 24933 ts_tablet_manager.cc:1398] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 10:33:05.426002 24933 tablet_bootstrap.cc:492] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef: Bootstrap starting.
I20240430 10:33:05.426012 24932 tablet_bootstrap.cc:492] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846: No bootstrap required, opened a new log
I20240430 10:33:05.426080 24932 ts_tablet_manager.cc:1367] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240430 10:33:05.426581 24932 raft_consensus.cc:348] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.426671 24932 raft_consensus.cc:374] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.426702 24932 raft_consensus.cc:729] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aa0f551289403caa641628dd390846, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.426831 24932 consensus_queue.cc:260] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.426920 24933 tablet_bootstrap.cc:654] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.427160 24932 ts_tablet_manager.cc:1398] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 10:33:05.427317 24932 tablet_bootstrap.cc:492] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846: Bootstrap starting.
I20240430 10:33:05.428082 24933 tablet_bootstrap.cc:492] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef: No bootstrap required, opened a new log
I20240430 10:33:05.428155 24933 ts_tablet_manager.cc:1367] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20240430 10:33:05.428267 24932 tablet_bootstrap.cc:654] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:05.428817 24933 raft_consensus.cc:348] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.428920 24933 raft_consensus.cc:374] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.428956 24933 raft_consensus.cc:729] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf979890494e40c8b3207effd7c83cef, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.429082 24933 consensus_queue.cc:260] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.429338 24933 ts_tablet_manager.cc:1398] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240430 10:33:05.429375 24932 tablet_bootstrap.cc:492] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846: No bootstrap required, opened a new log
I20240430 10:33:05.429420 24931 tablet_bootstrap.cc:492] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584: No bootstrap required, opened a new log
I20240430 10:33:05.429438 24932 ts_tablet_manager.cc:1367] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:33:05.429517 24931 ts_tablet_manager.cc:1367] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240430 10:33:05.429899 24931 raft_consensus.cc:348] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.429968 24932 raft_consensus.cc:348] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.430006 24931 raft_consensus.cc:374] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.430049 24931 raft_consensus.cc:729] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dffd8c702cd4fdba865ead4911f8584, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.430054 24932 raft_consensus.cc:374] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:05.430089 24932 raft_consensus.cc:729] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aa0f551289403caa641628dd390846, State: Initialized, Role: FOLLOWER
I20240430 10:33:05.430171 24931 consensus_queue.cc:260] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.430224 24932 consensus_queue.cc:260] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.430449 24931 ts_tablet_manager.cc:1398] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240430 10:33:05.430477 24932 ts_tablet_manager.cc:1398] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
W20240430 10:33:05.435225 24784 tablet.cc:2369] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:33:05.462808 24937 raft_consensus.cc:482] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:33:05.463045 24937 raft_consensus.cc:504] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.464195 24937 leader_election.cc:288] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:05.467995 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584" is_pre_election: true
I20240430 10:33:05.467994 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef" is_pre_election: true
I20240430 10:33:05.468271 24606 raft_consensus.cc:2454] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 0.
I20240430 10:33:05.468276 24868 raft_consensus.cc:2454] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 0.
I20240430 10:33:05.468683 24673 leader_election.cc:302] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf979890494e40c8b3207effd7c83cef, f2aa0f551289403caa641628dd390846; no voters: 
I20240430 10:33:05.468922 24937 raft_consensus.cc:2781] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:33:05.468986 24937 raft_consensus.cc:482] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:33:05.469018 24937 raft_consensus.cc:3037] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.469859 24937 raft_consensus.cc:504] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.470175 24937 leader_election.cc:288] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 election: Requested vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:05.470479 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584"
I20240430 10:33:05.470583 24868 raft_consensus.cc:3037] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.471556 24868 raft_consensus.cc:2454] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 1.
I20240430 10:33:05.471789 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef"
I20240430 10:33:05.471832 24672 leader_election.cc:302] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, f2aa0f551289403caa641628dd390846; no voters: 
I20240430 10:33:05.471910 24606 raft_consensus.cc:3037] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.472733 24606 raft_consensus.cc:2454] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 1.
I20240430 10:33:05.472930 24937 raft_consensus.cc:2781] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:33:05.473140 24937 raft_consensus.cc:686] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [term 1 LEADER]: Becoming Leader. State: Replica: f2aa0f551289403caa641628dd390846, State: Running, Role: LEADER
I20240430 10:33:05.473484 24937 consensus_queue.cc:237] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.476420 24463 catalog_manager.cc:5484] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2aa0f551289403caa641628dd390846 (127.23.217.66). New cstate: current_term: 1 leader_uuid: "f2aa0f551289403caa641628dd390846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } health_report { overall_health: UNKNOWN } } }
I20240430 10:33:05.489252 24939 raft_consensus.cc:482] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:33:05.489405 24939 raft_consensus.cc:504] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.490435 24939 leader_election.cc:288] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), f2aa0f551289403caa641628dd390846 (127.23.217.66:45035)
I20240430 10:33:05.494158 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebcf145701c453694b3c0e260fd396b" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef" is_pre_election: true
I20240430 10:33:05.494329 24606 raft_consensus.cc:2454] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dffd8c702cd4fdba865ead4911f8584 in term 0.
I20240430 10:33:05.494711 24804 leader_election.cc:302] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, cf979890494e40c8b3207effd7c83cef; no voters: 
I20240430 10:33:05.494733 24737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebcf145701c453694b3c0e260fd396b" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aa0f551289403caa641628dd390846" is_pre_election: true
I20240430 10:33:05.494940 24737 raft_consensus.cc:2454] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dffd8c702cd4fdba865ead4911f8584 in term 0.
I20240430 10:33:05.494963 24939 raft_consensus.cc:2781] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:33:05.495028 24939 raft_consensus.cc:482] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:33:05.495064 24939 raft_consensus.cc:3037] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.495968 24939 raft_consensus.cc:504] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.496268 24939 leader_election.cc:288] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 1 election: Requested vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), f2aa0f551289403caa641628dd390846 (127.23.217.66:45035)
I20240430 10:33:05.496562 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebcf145701c453694b3c0e260fd396b" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef"
I20240430 10:33:05.496595 24737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ebcf145701c453694b3c0e260fd396b" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aa0f551289403caa641628dd390846"
I20240430 10:33:05.496678 24606 raft_consensus.cc:3037] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.496694 24737 raft_consensus.cc:3037] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.497421 24606 raft_consensus.cc:2454] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dffd8c702cd4fdba865ead4911f8584 in term 1.
I20240430 10:33:05.497428 24737 raft_consensus.cc:2454] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dffd8c702cd4fdba865ead4911f8584 in term 1.
I20240430 10:33:05.497740 24803 leader_election.cc:302] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, f2aa0f551289403caa641628dd390846; no voters: 
W20240430 10:33:05.497848 24653 tablet.cc:2369] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:33:05.498392 24939 raft_consensus.cc:2781] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:33:05.498585 24939 raft_consensus.cc:686] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 LEADER]: Becoming Leader. State: Replica: 2dffd8c702cd4fdba865ead4911f8584, State: Running, Role: LEADER
I20240430 10:33:05.498880 24939 consensus_queue.cc:237] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.501791 24463 catalog_manager.cc:5484] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67). New cstate: current_term: 1 leader_uuid: "2dffd8c702cd4fdba865ead4911f8584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } health_report { overall_health: HEALTHY } } }
W20240430 10:33:05.599547 24915 tablet.cc:2369] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:33:05.607637 24937 raft_consensus.cc:482] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:33:05.607771 24937 raft_consensus.cc:504] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.608237 24937 leader_election.cc:288] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:05.608711 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584" is_pre_election: true
I20240430 10:33:05.608721 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef" is_pre_election: true
I20240430 10:33:05.608834 24868 raft_consensus.cc:2454] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 0.
I20240430 10:33:05.608870 24606 raft_consensus.cc:2454] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 0.
I20240430 10:33:05.609138 24672 leader_election.cc:302] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, f2aa0f551289403caa641628dd390846; no voters: 
I20240430 10:33:05.609313 24937 raft_consensus.cc:2781] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:33:05.609372 24937 raft_consensus.cc:482] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:33:05.609400 24937 raft_consensus.cc:3037] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.610165 24937 raft_consensus.cc:504] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.610440 24937 leader_election.cc:288] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 election: Requested vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:05.610774 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef"
I20240430 10:33:05.610814 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "f2aa0f551289403caa641628dd390846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584"
I20240430 10:33:05.610882 24606 raft_consensus.cc:3037] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.610975 24868 raft_consensus.cc:3037] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:05.611646 24606 raft_consensus.cc:2454] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 1.
I20240430 10:33:05.611680 24868 raft_consensus.cc:2454] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aa0f551289403caa641628dd390846 in term 1.
I20240430 10:33:05.611933 24672 leader_election.cc:302] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, f2aa0f551289403caa641628dd390846; no voters: 
I20240430 10:33:05.612146 24937 raft_consensus.cc:2781] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:33:05.612241 24937 raft_consensus.cc:686] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [term 1 LEADER]: Becoming Leader. State: Replica: f2aa0f551289403caa641628dd390846, State: Running, Role: LEADER
I20240430 10:33:05.612390 24937 consensus_queue.cc:237] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:05.614035 24463 catalog_manager.cc:5484] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2aa0f551289403caa641628dd390846 (127.23.217.66). New cstate: current_term: 1 leader_uuid: "f2aa0f551289403caa641628dd390846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } health_report { overall_health: UNKNOWN } } }
I20240430 10:33:05.632755 24421 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 10:33:05.634563 24421 ts_itest-base.cc:233] Waiting for 3 tablets on tserver cf979890494e40c8b3207effd7c83cef to finish bootstrapping
I20240430 10:33:05.640506 24421 ts_itest-base.cc:233] Waiting for 3 tablets on tserver f2aa0f551289403caa641628dd390846 to finish bootstrapping
I20240430 10:33:05.645488 24421 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 2dffd8c702cd4fdba865ead4911f8584 to finish bootstrapping
I20240430 10:33:05.651046 24954 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
  http://127.23.217.67:44449/startup
  http://127.23.217.65:36497/transactions?tablet_id=e1b2f890834447e1b34a35e5bcf1203b
  http://127.23.217.65:36497/tablets
  http://127.23.217.65:36497/mem-trackers
  http://127.23.217.67:44449/stacks
  http://127.23.217.67:44449/maintenance-manager
  http://127.23.217.66:42135/threadz
  http://127.23.217.126:44427/threadz?group=all
  http://127.23.217.67:44449/metrics
  http://127.23.217.67:44449/threadz
  http://127.23.217.66:42135/metrics
  http://127.23.217.66:42135/transactions?tablet_id=e1b2f890834447e1b34a35e5bcf1203b
  http://127.23.217.66:42135/mem-trackers
  http://127.23.217.65:36497/metrics
  http://127.23.217.126:44427/mem-trackers
  http://127.23.217.65:36497/maintenance-manager
  http://127.23.217.66:42135/scans
  http://127.23.217.126:44427/ipki-ca-cert-pem
  http://127.23.217.126:44427/threadz
  http://127.23.217.126:44427/dump-entities
  http://127.23.217.66:42135/startup
  http://127.23.217.66:42135/tablets
  http://127.23.217.126:44427/tablet-servers
  http://127.23.217.66:42135/stacks
  http://127.23.217.67:44449/tablets
  http://127.23.217.65:36497/stacks
  http://127.23.217.67:44449/transactions?tablet_id=e1b2f890834447e1b34a35e5bcf1203b
  http://127.23.217.67:44449/mem-trackers
  http://127.23.217.65:36497/threadz
  http://127.23.217.126:44427/startup
  http://127.23.217.67:44449/scans
  http://127.23.217.65:36497/threadz?group=all
  http://127.23.217.126:44427/ipki-ca-cert-der
  http://127.23.217.66:42135/threadz?group=all
  http://127.23.217.65:36497/startup
  http://127.23.217.126:44427/stacks
  http://127.23.217.126:44427/metrics
  http://127.23.217.126:44427/masters
  http://127.23.217.126:44427/tables
  http://127.23.217.126:44427/ipki-ca-cert
  http://127.23.217.65:36497/scans
  http://127.23.217.66:42135/maintenance-manager
  http://127.23.217.67:44449/threadz?group=all
I20240430 10:33:05.653952 24421 linked_list-test-util.h:401] Running for: 30.000s
I20240430 10:33:05.654018 24421 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 10:33:05.654043 24421 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
I20240430 10:33:05.678215 24606 raft_consensus.cc:1261] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Refusing update from remote peer f2aa0f551289403caa641628dd390846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:33:05.678642 24605 raft_consensus.cc:1261] T 6ebcf145701c453694b3c0e260fd396b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Refusing update from remote peer 2dffd8c702cd4fdba865ead4911f8584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:33:05.678840 24867 raft_consensus.cc:1261] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Refusing update from remote peer f2aa0f551289403caa641628dd390846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:33:05.678946 24868 raft_consensus.cc:1261] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Refusing update from remote peer f2aa0f551289403caa641628dd390846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:33:05.679746 24737 raft_consensus.cc:1261] T 6ebcf145701c453694b3c0e260fd396b P f2aa0f551289403caa641628dd390846 [term 1 FOLLOWER]: Refusing update from remote peer 2dffd8c702cd4fdba865ead4911f8584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:33:05.679700 24937 consensus_queue.cc:1035] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:33:05.680039 24947 consensus_queue.cc:1035] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:33:05.680768 24939 consensus_queue.cc:1035] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:33:05.680853 24605 raft_consensus.cc:1261] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Refusing update from remote peer f2aa0f551289403caa641628dd390846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 10:33:05.681485 24947 consensus_queue.cc:1035] T 07177c6fdc5742458352ef97b02bebb9 P f2aa0f551289403caa641628dd390846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:33:05.681579 24937 consensus_queue.cc:1035] T e1b2f890834447e1b34a35e5bcf1203b P f2aa0f551289403caa641628dd390846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:33:05.683466 24950 consensus_queue.cc:1035] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 10:33:05.693078 24970 mvcc.cc:205] Tried to move back new op lower bound from 7022482168530432000 to 7022482167806095360. Current Snapshot: MvccSnapshot[applied={T|T < 7022482168530432000 or (T in {7022482168530432000})}]
I20240430 10:33:05.694370 24962 mvcc.cc:205] Tried to move back new op lower bound from 7022482168529555456 to 7022482168269471744. Current Snapshot: MvccSnapshot[applied={T|T < 7022482168529555456}]
I20240430 10:33:05.698990 24969 mvcc.cc:205] Tried to move back new op lower bound from 7022482168529555456 to 7022482168269471744. Current Snapshot: MvccSnapshot[applied={T|T < 7022482168529555456}]
I20240430 10:33:06.438969 24990 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:07.780741 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:08.445446 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:09.112823 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
F20240430 10:33:09.157382 24990 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
    @     0x7f845a114dcd  google::LogMessage::Fail() at ??:0
    @     0x7f845a118b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f845a1147cc  google::LogMessage::Flush() at ??:0
    @     0x7f845a115f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f845b4f65b1  kudu::operator-() at ??:0
    @     0x7f845d5d9fd2  kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7f845d5c305b  kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7f845d5c2f90  kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7f845d5999bd  kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7f845e1c585d  kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
I20240430 10:33:09.194701 24784 maintenance_manager.cc:419] P f2aa0f551289403caa641628dd390846: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
    @     0x7f845e1c4936  _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
    @     0x7f845e1cd145  _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
    @     0x7f845908fd09  std::function<>::operator()() at ??:0
    @     0x7f845908a107  kudu::Webserver::RunPathHandler() at ??:0
    @     0x7f8459089a38  kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7f8459089123  kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7f84590b1b89  handle_request at ??:0
    @     0x7f84590b426b  worker_thread at ??:0
    @     0x7f845bd936db  start_thread at ??:0
    @     0x7f84592cb71f  clone at ??:0
I20240430 10:33:09.254050 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:09.421962 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.309s	user 0.258s	sys 0.012s Metrics: {"bytes_written":145021,"cfile_init":1,"compiler_manager_pool.queue_time_us":1279,"dirs.queue_time_us":16201,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":992,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1892,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7896,"thread_start_us":182,"threads_started":2}
I20240430 10:33:09.423210 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 134559 bytes on disk
I20240430 10:33:09.423933 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240430 10:33:09.535847 24919 meta_cache.cc:302] tablet 07177c6fdc5742458352ef97b02bebb9: replica f2aa0f551289403caa641628dd390846 (127.23.217.66:45035) has failed: Network error: recv got EOF from 127.23.217.66:45035 (error 108)
W20240430 10:33:09.550401 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.558352 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
I20240430 10:33:09.570797 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.316s	user 0.257s	sys 0.004s Metrics: {"bytes_written":146725,"cfile_init":1,"compiler_manager_pool.queue_time_us":169,"dirs.queue_time_us":245,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":996,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2077,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7996,"thread_start_us":213,"threads_started":2}
I20240430 10:33:09.571949 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 135502 bytes on disk
I20240430 10:33:09.572568 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240430 10:33:09.573474 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.584980 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.604841 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.617259 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.639744 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.655207 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.691109 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.702705 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
W20240430 10:33:09.703588 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 10:33:09.706604 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.744321 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.766122 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.813710 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.817433 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.822782 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.824085 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.826200 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.833181 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.835347 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.841718 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.851688 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.864782 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.875452 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.880849 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.887384 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.903103 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.915083 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.917026 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.922225 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.959127 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.966279 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:09.971791 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:09.979187 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.000855 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.019093 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.028259 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
I20240430 10:33:10.033344 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:10.038146 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.086367 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.095860 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.097990 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.112231 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.130189 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.163642 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.191540 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.198940 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.206326 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.237309 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.249594 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.263000 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.279680 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.315253 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.343566 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.353210 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.377246 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.390815 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.436877 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
I20240430 10:33:10.439311 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:10.448173 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.465451 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.478885 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.486017 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.543098 24919 meta_cache.cc:302] tablet e1b2f890834447e1b34a35e5bcf1203b: replica f2aa0f551289403caa641628dd390846 (127.23.217.66:45035) has failed: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
W20240430 10:33:10.557461 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.566597 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.579218 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
W20240430 10:33:10.594796 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.614019 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
W20240430 10:33:10.619189 24828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47076: Illegal state: replica 2dffd8c702cd4fdba865ead4911f8584 is not leader of this config: current role FOLLOWER
I20240430 10:33:10.654067 25035 raft_consensus.cc:482] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2aa0f551289403caa641628dd390846)
I20240430 10:33:10.654223 25035 raft_consensus.cc:504] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.654647 25039 raft_consensus.cc:482] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2aa0f551289403caa641628dd390846)
I20240430 10:33:10.654721 25035 leader_election.cc:288] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), f2aa0f551289403caa641628dd390846 (127.23.217.66:45035)
I20240430 10:33:10.654814 25039 raft_consensus.cc:504] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.655249 24605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 2 candidate_status { last_received { term: 1 index: 243 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef" is_pre_election: true
I20240430 10:33:10.655427 24605 raft_consensus.cc:2454] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dffd8c702cd4fdba865ead4911f8584 in term 1.
I20240430 10:33:10.656004 24804 leader_election.cc:302] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, cf979890494e40c8b3207effd7c83cef; no voters: 
I20240430 10:33:10.656220 25035 raft_consensus.cc:2781] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 10:33:10.656292 25035 raft_consensus.cc:482] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f2aa0f551289403caa641628dd390846)
I20240430 10:33:10.656351 25035 raft_consensus.cc:3037] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Advancing to term 2
W20240430 10:33:10.657277 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
W20240430 10:33:10.657493 24803 leader_election.cc:334] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
I20240430 10:33:10.657716 25035 raft_consensus.cc:504] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.657889 25039 leader_election.cc:288] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2aa0f551289403caa641628dd390846 (127.23.217.66:45035), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:10.658100 25035 leader_election.cc:288] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 election: Requested vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), f2aa0f551289403caa641628dd390846 (127.23.217.66:45035)
I20240430 10:33:10.658669 24605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 2 candidate_status { last_received { term: 1 index: 243 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef"
I20240430 10:33:10.658823 24605 raft_consensus.cc:3037] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Advancing to term 2
W20240430 10:33:10.659165 24803 leader_election.cc:334] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
W20240430 10:33:10.660004 24541 leader_election.cc:334] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
I20240430 10:33:10.660179 24605 raft_consensus.cc:2454] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dffd8c702cd4fdba865ead4911f8584 in term 2.
I20240430 10:33:10.660717 24804 leader_election.cc:302] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, cf979890494e40c8b3207effd7c83cef; no voters: f2aa0f551289403caa641628dd390846
I20240430 10:33:10.660933 25035 raft_consensus.cc:2781] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 10:33:10.661031 25035 raft_consensus.cc:686] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [term 2 LEADER]: Becoming Leader. State: Replica: 2dffd8c702cd4fdba865ead4911f8584, State: Running, Role: LEADER
I20240430 10:33:10.661170 25035 consensus_queue.cc:237] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.243, Last appended by leader: 243, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.661526 24950 raft_consensus.cc:482] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2aa0f551289403caa641628dd390846)
I20240430 10:33:10.661617 24950 raft_consensus.cc:504] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.662019 24950 leader_election.cc:288] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf979890494e40c8b3207effd7c83cef (127.23.217.65:34067), f2aa0f551289403caa641628dd390846 (127.23.217.66:45035)
I20240430 10:33:10.662422 24605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "2dffd8c702cd4fdba865ead4911f8584" candidate_term: 2 candidate_status { last_received { term: 1 index: 240 } } ignore_live_leader: false dest_uuid: "cf979890494e40c8b3207effd7c83cef" is_pre_election: true
W20240430 10:33:10.663772 24803 leader_election.cc:334] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
I20240430 10:33:10.663847 24803 leader_election.cc:302] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584; no voters: cf979890494e40c8b3207effd7c83cef, f2aa0f551289403caa641628dd390846
I20240430 10:33:10.664013 24459 catalog_manager.cc:5484] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 reported cstate change: term changed from 1 to 2, leader changed from f2aa0f551289403caa641628dd390846 (127.23.217.66) to 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67). New cstate: current_term: 2 leader_uuid: "2dffd8c702cd4fdba865ead4911f8584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } health_report { overall_health: HEALTHY } } }
I20240430 10:33:10.664490 24950 raft_consensus.cc:2726] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:33:10.664793 25039 raft_consensus.cc:482] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2aa0f551289403caa641628dd390846)
I20240430 10:33:10.664836 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07177c6fdc5742458352ef97b02bebb9" candidate_uuid: "cf979890494e40c8b3207effd7c83cef" candidate_term: 2 candidate_status { last_received { term: 1 index: 243 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584" is_pre_election: true
I20240430 10:33:10.664983 25039 raft_consensus.cc:504] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.665272 24541 leader_election.cc:302] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf979890494e40c8b3207effd7c83cef; no voters: 2dffd8c702cd4fdba865ead4911f8584, f2aa0f551289403caa641628dd390846
I20240430 10:33:10.665342 25039 leader_election.cc:288] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2aa0f551289403caa641628dd390846 (127.23.217.66:45035), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:10.665510 25039 raft_consensus.cc:2726] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:33:10.666144 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "cf979890494e40c8b3207effd7c83cef" candidate_term: 2 candidate_status { last_received { term: 1 index: 240 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584" is_pre_election: true
I20240430 10:33:10.666266 24867 raft_consensus.cc:2454] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf979890494e40c8b3207effd7c83cef in term 1.
I20240430 10:33:10.666592 24541 leader_election.cc:302] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, cf979890494e40c8b3207effd7c83cef; no voters: 
I20240430 10:33:10.666760 25039 raft_consensus.cc:2781] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 10:33:10.666826 25039 raft_consensus.cc:482] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Starting leader election (detected failure of leader f2aa0f551289403caa641628dd390846)
I20240430 10:33:10.666868 25039 raft_consensus.cc:3037] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 1 FOLLOWER]: Advancing to term 2
W20240430 10:33:10.667181 24541 leader_election.cc:334] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
I20240430 10:33:10.667791 25039 raft_consensus.cc:504] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
I20240430 10:33:10.668103 25039 leader_election.cc:288] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 election: Requested vote from peers f2aa0f551289403caa641628dd390846 (127.23.217.66:45035), 2dffd8c702cd4fdba865ead4911f8584 (127.23.217.67:41777)
I20240430 10:33:10.668555 24867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2f890834447e1b34a35e5bcf1203b" candidate_uuid: "cf979890494e40c8b3207effd7c83cef" candidate_term: 2 candidate_status { last_received { term: 1 index: 240 } } ignore_live_leader: false dest_uuid: "2dffd8c702cd4fdba865ead4911f8584"
I20240430 10:33:10.668653 24867 raft_consensus.cc:3037] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 1 FOLLOWER]: Advancing to term 2
I20240430 10:33:10.669582 24867 raft_consensus.cc:2454] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf979890494e40c8b3207effd7c83cef in term 2.
I20240430 10:33:10.669850 24541 leader_election.cc:302] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dffd8c702cd4fdba865ead4911f8584, cf979890494e40c8b3207effd7c83cef; no voters: 
W20240430 10:33:10.670033 24541 leader_election.cc:334] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
I20240430 10:33:10.670083 25039 raft_consensus.cc:2781] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 2 FOLLOWER]: Leader election won for term 2
I20240430 10:33:10.670281 25039 raft_consensus.cc:686] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [term 2 LEADER]: Becoming Leader. State: Replica: cf979890494e40c8b3207effd7c83cef, State: Running, Role: LEADER
I20240430 10:33:10.670653 25039 consensus_queue.cc:237] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 240, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } }
W20240430 10:33:10.673005 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
I20240430 10:33:10.673157 24459 catalog_manager.cc:5484] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef reported cstate change: term changed from 1 to 2, leader changed from f2aa0f551289403caa641628dd390846 (127.23.217.66) to cf979890494e40c8b3207effd7c83cef (127.23.217.65). New cstate: current_term: 2 leader_uuid: "cf979890494e40c8b3207effd7c83cef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2aa0f551289403caa641628dd390846" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 } health_report { overall_health: UNKNOWN } } }
W20240430 10:33:10.710436 24566 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55412: Illegal state: replica cf979890494e40c8b3207effd7c83cef is not leader of this config: current role FOLLOWER
I20240430 10:33:10.717602 24605 raft_consensus.cc:1261] T 07177c6fdc5742458352ef97b02bebb9 P cf979890494e40c8b3207effd7c83cef [term 2 FOLLOWER]: Refusing update from remote peer 2dffd8c702cd4fdba865ead4911f8584: Log matching property violated. Preceding OpId in replica: term: 1 index: 243. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
W20240430 10:33:10.718068 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 10:33:10.718111 24950 consensus_queue.cc:1035] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf979890494e40c8b3207effd7c83cef" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20240430 10:33:10.749231 24867 raft_consensus.cc:1261] T e1b2f890834447e1b34a35e5bcf1203b P 2dffd8c702cd4fdba865ead4911f8584 [term 2 FOLLOWER]: Refusing update from remote peer cf979890494e40c8b3207effd7c83cef: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20240430 10:33:10.749883 25039 consensus_queue.cc:1035] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dffd8c702cd4fdba865ead4911f8584" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
W20240430 10:33:10.756088 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 10:33:11.666600 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:12.232237 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:12.572209 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 10:33:13.109086 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240430 10:33:13.346874 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 10:33:13.434228 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:13.580466 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:13.681890 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.247s	user 0.210s	sys 0.000s Metrics: {"bytes_written":147159,"cfile_init":1,"dirs.queue_time_us":24483,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":937,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1355,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8028,"thread_start_us":73,"threads_started":1}
I20240430 10:33:13.683156 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 126698 bytes on disk
I20240430 10:33:13.683756 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240430 10:33:13.684403 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014689
I20240430 10:33:13.708528 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:13.908516 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.328s	user 0.210s	sys 0.008s Metrics: {"bytes_written":150540,"cfile_init":1,"dirs.queue_time_us":13515,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":774,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1433,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8287,"thread_start_us":190,"threads_started":1}
I20240430 10:33:13.909590 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 128547 bytes on disk
I20240430 10:33:13.925160 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":337,"lbm_reads_lt_1ms":4}
I20240430 10:33:13.930804 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014822
I20240430 10:33:14.270751 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:14.780346 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240430 10:33:14.991402 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.306s	user 0.672s	sys 0.004s Metrics: {"bytes_written":12123,"cfile_cache_hit":13,"cfile_cache_hit_bytes":140587,"cfile_cache_miss":55,"cfile_cache_miss_bytes":882089,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":1154,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":10385,"drs_written":1,"lbm_read_time_us":1934,"lbm_reads_lt_1ms":83,"lbm_write_time_us":2451,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":477228,"reinsert_count":0,"rows_written":15924,"thread_start_us":542,"threads_started":8,"update_count":1873}
I20240430 10:33:14.992184 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 277435 bytes on disk
I20240430 10:33:14.992647 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240430 10:33:14.993212 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.048374
W20240430 10:33:15.104043 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 10:33:15.105100 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.112s	user 0.063s	sys 0.004s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14500,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":371,"lbm_reads_lt_1ms":21,"lbm_write_time_us":261,"lbm_writes_lt_1ms":9,"reinsert_count":0,"update_count":1873}
I20240430 10:33:15.249641 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.319s	user 0.411s	sys 0.007s Metrics: {"bytes_written":14394,"cfile_cache_hit":13,"cfile_cache_hit_bytes":143293,"cfile_cache_miss":55,"cfile_cache_miss_bytes":895137,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":4685,"dirs.run_cpu_time_us":834,"dirs.run_wall_time_us":8633,"drs_written":1,"lbm_read_time_us":1940,"lbm_reads_lt_1ms":83,"lbm_write_time_us":2813,"lbm_writes_lt_1ms":55,"mutex_wait_us":3510,"num_input_rowsets":2,"peak_mem_usage":480998,"reinsert_count":0,"rows_written":16283,"thread_start_us":431,"threads_started":6,"update_count":2232}
I20240430 10:33:15.250411 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.056207
I20240430 10:33:15.305759 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.053s	user 0.051s	sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":16843,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":43,"lbm_read_time_us":316,"lbm_reads_lt_1ms":25,"lbm_write_time_us":1807,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":8,"mutex_wait_us":25,"reinsert_count":0,"spinlock_wait_cycles":896,"update_count":2232}
I20240430 10:33:15.306356 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 281465 bytes on disk
I20240430 10:33:15.306793 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240430 10:33:15.605734 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240430 10:33:15.654217 24421 linked_list-test-util.h:417] Sample at HT timestamp: 7022482209371275264 Inserted count: 30150
I20240430 10:33:15.852710 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:15.869105 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 10:33:16.227046 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240430 10:33:16.411438 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:17.109387 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:17.310819 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
W20240430 10:33:17.445235 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 10:33:17.446189 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.337s	user 0.296s	sys 0.000s Metrics: {"bytes_written":152204,"cfile_init":1,"dirs.queue_time_us":14630,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1972,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1691,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8386,"thread_start_us":95,"threads_started":1}
I20240430 10:33:17.447235 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014703
I20240430 10:33:17.729771 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.419s	user 0.235s	sys 0.003s Metrics: {"bytes_written":159537,"cfile_init":1,"dirs.queue_time_us":16640,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":916,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4319,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8567,"thread_start_us":98,"threads_started":1}
I20240430 10:33:17.730527 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 147116 bytes on disk
I20240430 10:33:17.731349 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240430 10:33:17.731920 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014698
I20240430 10:33:17.915082 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:18.159173 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240430 10:33:18.269549 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 10:33:18.483381 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:18.549175 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.102s	user 0.935s	sys 0.008s Metrics: {"bytes_written":1275,"cfile_cache_hit":13,"cfile_cache_hit_bytes":141820,"cfile_cache_miss":79,"cfile_cache_miss_bytes":1491372,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1691,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":3932,"drs_written":1,"lbm_read_time_us":3536,"lbm_reads_lt_1ms":115,"lbm_write_time_us":4083,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":801174,"reinsert_count":0,"rows_written":24310,"thread_start_us":499,"threads_started":7,"update_count":167}
I20240430 10:33:18.550120 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.003379
I20240430 10:33:18.599233 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.049s	user 0.040s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1918,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":36,"lbm_read_time_us":297,"lbm_reads_lt_1ms":24,"lbm_write_time_us":199,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":167}
I20240430 10:33:18.599920 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 456576 bytes on disk
I20240430 10:33:18.601260 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240430 10:33:18.716976 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.985s	user 0.569s	sys 0.000s Metrics: {"bytes_written":4725,"cfile_cache_hit":12,"cfile_cache_hit_bytes":51046,"cfile_cache_miss":80,"cfile_cache_miss_bytes":1609784,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":10582,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":5777,"drs_written":1,"lbm_read_time_us":2689,"lbm_reads_lt_1ms":112,"lbm_write_time_us":3757,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":809578,"reinsert_count":0,"rows_written":24850,"thread_start_us":475,"threads_started":7,"update_count":707}
I20240430 10:33:18.718011 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.012207
I20240430 10:33:18.772506 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.054s	user 0.045s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":6751,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":62,"lbm_read_time_us":312,"lbm_reads_lt_1ms":24,"lbm_write_time_us":249,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":707}
I20240430 10:33:18.773169 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 462253 bytes on disk
I20240430 10:33:18.773700 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240430 10:33:19.855260 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(6ebcf145701c453694b3c0e260fd396b): perf score=1.000000
I20240430 10:33:19.936759 25136 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 21 ops behind the committed index 
W20240430 10:33:19.954033 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 10:33:20.005427 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 10:33:20.026326 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(6ebcf145701c453694b3c0e260fd396b): perf score=1.000000
I20240430 10:33:20.088574 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.233s	user 0.194s	sys 0.004s Metrics: {"bytes_written":144736,"cfile_init":1,"dirs.queue_time_us":12179,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":956,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1304,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7802,"thread_start_us":128,"threads_started":1}
I20240430 10:33:20.089246 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=1.000000
W20240430 10:33:20.234036 24957 debug-util.cc:398] Leaking SignalData structure 0x55feac694840 after lost signal to thread 25120
I20240430 10:33:20.320984 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.230s	user 0.228s	sys 0.000s Metrics: {"bytes_written":5815,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":945,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1636,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7885,"spinlock_wait_cycles":12160,"update_count":879}
I20240430 10:33:20.322355 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b): 143069 bytes on disk
I20240430 10:33:20.332140 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240430 10:33:20.334133 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b): 142429 bytes on disk
I20240430 10:33:20.334939 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240430 10:33:20.335705 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=0.046436
I20240430 10:33:20.365180 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.339s	user 0.219s	sys 0.004s Metrics: {"bytes_written":5926,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9066,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":777,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9549,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7940,"thread_start_us":94,"threads_started":1,"update_count":896}
I20240430 10:33:20.366217 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b): 143845 bytes on disk
I20240430 10:33:20.366727 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240430 10:33:20.367296 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=1.000000
I20240430 10:33:20.433753 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.097s	user 0.025s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":6944,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":38,"lbm_read_time_us":292,"lbm_reads_lt_1ms":20,"lbm_write_time_us":253,"lbm_writes_lt_1ms":8,"reinsert_count":0,"thread_start_us":110,"threads_started":1,"update_count":879}
I20240430 10:33:20.434444 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:20.702576 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.268s	user 0.214s	sys 0.008s Metrics: {"bytes_written":145731,"cfile_init":1,"dirs.queue_time_us":2234,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":825,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1487,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7947}
I20240430 10:33:20.703644 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 135898 bytes on disk
I20240430 10:33:20.707178 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240430 10:33:20.707863 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014573
I20240430 10:33:20.745653 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.368s	user 0.244s	sys 0.000s Metrics: {"bytes_written":147137,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":759,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2261,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8012}
I20240430 10:33:20.747630 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b): 151821 bytes on disk
I20240430 10:33:20.749768 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240430 10:33:20.750455 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
W20240430 10:33:21.014564 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 10:33:21.028749 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.277s	user 0.223s	sys 0.000s Metrics: {"bytes_written":148402,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":804,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1458,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8117}
I20240430 10:33:21.030143 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 133094 bytes on disk
I20240430 10:33:21.031234 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":768}
I20240430 10:33:21.033314 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014568
W20240430 10:33:21.074667 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 10:33:21.355731 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:21.599328 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240430 10:33:21.982596 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.274s	user 0.762s	sys 0.004s Metrics: {"bytes_written":4281,"cfile_cache_hit":12,"cfile_cache_hit_bytes":47705,"cfile_cache_miss":94,"cfile_cache_miss_bytes":2141395,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":5726,"dirs.run_cpu_time_us":274,"dirs.run_wall_time_us":4915,"drs_written":1,"lbm_read_time_us":3576,"lbm_reads_lt_1ms":126,"lbm_write_time_us":15736,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":78,"num_input_rowsets":2,"peak_mem_usage":1058941,"reinsert_count":0,"rows_written":32257,"spinlock_wait_cycles":768,"thread_start_us":639,"threads_started":8,"update_count":638}
I20240430 10:33:21.983878 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.008430
I20240430 10:33:22.117182 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.133s	user 0.080s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":6350,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":88,"lbm_read_time_us":593,"lbm_reads_lt_1ms":24,"lbm_write_time_us":278,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":638}
I20240430 10:33:22.118046 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 610180 bytes on disk
I20240430 10:33:22.118937 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":332,"lbm_reads_lt_1ms":4}
I20240430 10:33:22.151394 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:22.391199 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 10:33:22.533435 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.490s	user 0.763s	sys 0.016s Metrics: {"bytes_written":8755,"cfile_cache_hit":12,"cfile_cache_hit_bytes":48699,"cfile_cache_miss":94,"cfile_cache_miss_bytes":2175075,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1891,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1164,"drs_written":1,"lbm_read_time_us":3223,"lbm_reads_lt_1ms":126,"lbm_write_time_us":5676,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":83,"num_input_rowsets":2,"peak_mem_usage":1069935,"reinsert_count":0,"rows_written":32967,"thread_start_us":705,"threads_started":8,"update_count":1348}
I20240430 10:33:22.534260 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(6ebcf145701c453694b3c0e260fd396b): perf score=0.046959
I20240430 10:33:22.560441 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.026s	user 0.024s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7076,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":71,"lbm_read_time_us":250,"lbm_reads_lt_1ms":20,"lbm_write_time_us":220,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":896}
I20240430 10:33:22.560977 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 618045 bytes on disk
I20240430 10:33:22.561455 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240430 10:33:22.561980 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.016830
I20240430 10:33:22.643378 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.081s	user 0.067s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":12135,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":62,"lbm_read_time_us":264,"lbm_reads_lt_1ms":20,"lbm_write_time_us":250,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1348}
I20240430 10:33:22.946811 25035 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 212 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:23.526604 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:23.557468 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240430 10:33:23.613044 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 10:33:24.063659 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:24.125823 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:24.507238 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.381s	user 0.291s	sys 0.004s Metrics: {"bytes_written":17583,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9974,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":4647,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1783,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8562,"thread_start_us":65,"threads_started":1,"update_count":2738}
I20240430 10:33:24.508251 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 130513 bytes on disk
I20240430 10:33:24.511641 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240430 10:33:24.512344 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014453
I20240430 10:33:24.647250 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:24.955359 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.308s	user 0.245s	sys 0.000s Metrics: {"bytes_written":165085,"cfile_init":1,"dirs.queue_time_us":13823,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1130,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1670,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":8921,"thread_start_us":291,"threads_started":1}
I20240430 10:33:24.956219 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 153089 bytes on disk
I20240430 10:33:24.956692 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240430 10:33:24.957319 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014440
W20240430 10:33:25.036003 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240430 10:33:25.036773 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240430 10:33:25.657640 24421 linked_list-test-util.h:417] Sample at HT timestamp: 7022482250341498880 Inserted count: 65450
I20240430 10:33:25.763775 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.248s	user 1.015s	sys 0.004s Metrics: {"bytes_written":736877,"cfile_cache_hit":14,"cfile_cache_hit_bytes":364450,"cfile_cache_miss":114,"cfile_cache_miss_bytes":2439472,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":1243,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":3350,"drs_written":1,"lbm_read_time_us":3972,"lbm_reads_lt_1ms":150,"lbm_write_time_us":5218,"lbm_writes_lt_1ms":93,"num_input_rowsets":2,"peak_mem_usage":1357145,"rows_written":40819,"thread_start_us":527,"threads_started":8}
I20240430 10:33:25.764627 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 787289 bytes on disk
I20240430 10:33:25.766474 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240430 10:33:25.956986 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:26.069530 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240430 10:33:26.236361 25091 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 441 ops behind the committed index  [suppressed 5 similar messages]
W20240430 10:33:26.274339 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240430 10:33:26.364089 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.407s	user 1.076s	sys 0.000s Metrics: {"bytes_written":5514,"cfile_cache_hit":14,"cfile_cache_hit_bytes":421959,"cfile_cache_miss":116,"cfile_cache_miss_bytes":2450129,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1272,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1013,"drs_written":1,"lbm_read_time_us":4223,"lbm_reads_lt_1ms":148,"lbm_write_time_us":5940,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1378067,"reinsert_count":0,"rows_written":41888,"spinlock_wait_cycles":1920,"thread_start_us":782,"threads_started":9,"update_count":831}
I20240430 10:33:26.364850 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.008392
I20240430 10:33:26.503533 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.135s	user 0.067s	sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":8307,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":323,"lbm_reads_lt_1ms":24,"lbm_write_time_us":221,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":831}
I20240430 10:33:26.504696 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 801025 bytes on disk
I20240430 10:33:26.505146 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240430 10:33:26.561937 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:26.626971 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240430 10:33:27.751837 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 10:33:27.781075 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:27.869141 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:28.117605 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.336s	user 0.323s	sys 0.000s Metrics: {"bytes_written":149874,"cfile_init":1,"dirs.queue_time_us":257,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":883,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2136,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8143,"thread_start_us":126,"threads_started":1}
I20240430 10:33:28.118333 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 135614 bytes on disk
I20240430 10:33:28.118745 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240430 10:33:28.119256 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014448
I20240430 10:33:28.346630 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:28.646816 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 10:33:28.760175 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
W20240430 10:33:28.882834 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 10:33:29.075068 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.315s	user 0.219s	sys 0.001s Metrics: {"bytes_written":147158,"cfile_init":1,"dirs.queue_time_us":7029,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":4747,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1476,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7975,"thread_start_us":78,"threads_started":1}
I20240430 10:33:29.075978 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014365
I20240430 10:33:29.362357 25235 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 620 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:29.571853 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.452s	user 1.391s	sys 0.012s Metrics: {"bytes_written":7257,"cfile_cache_hit":13,"cfile_cache_hit_bytes":139091,"cfile_cache_miss":129,"cfile_cache_miss_bytes":3239487,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":10784,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":7971,"drs_written":1,"lbm_read_time_us":4785,"lbm_reads_lt_1ms":157,"lbm_write_time_us":6957,"lbm_writes_lt_1ms":110,"num_input_rowsets":2,"peak_mem_usage":1610144,"reinsert_count":0,"rows_written":48962,"thread_start_us":477,"threads_started":7,"update_count":1109}
I20240430 10:33:29.572724 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.009459
I20240430 10:33:29.700927 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.127s	user 0.098s	sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":10853,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":204,"lbm_read_time_us":363,"lbm_reads_lt_1ms":24,"lbm_write_time_us":254,"lbm_writes_lt_1ms":8,"mutex_wait_us":119,"reinsert_count":0,"update_count":1109}
I20240430 10:33:29.701676 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 939955 bytes on disk
I20240430 10:33:29.702116 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240430 10:33:30.320662 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240430 10:33:30.354959 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:30.443845 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240430 10:33:30.543746 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.468s	user 1.279s	sys 0.004s Metrics: {"bytes_written":4907,"cfile_cache_hit":14,"cfile_cache_hit_bytes":413520,"cfile_cache_miss":134,"cfile_cache_miss_bytes":3030940,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":8072,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":865,"drs_written":1,"lbm_read_time_us":4438,"lbm_reads_lt_1ms":170,"lbm_write_time_us":7118,"lbm_writes_lt_1ms":112,"num_input_rowsets":2,"peak_mem_usage":1646881,"reinsert_count":0,"rows_written":49863,"thread_start_us":542,"threads_started":7,"update_count":736}
I20240430 10:33:30.544955 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.006289
I20240430 10:33:30.713841 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.169s	user 0.118s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":7575,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":34,"lbm_read_time_us":370,"lbm_reads_lt_1ms":24,"lbm_write_time_us":239,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":736}
I20240430 10:33:30.714736 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 961614 bytes on disk
I20240430 10:33:30.721468 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240430 10:33:30.930428 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:31.312094 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 10:33:31.367167 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240430 10:33:31.815527 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240430 10:33:32.219918 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:32.336803 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:32.488850 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.269s	user 0.227s	sys 0.000s Metrics: {"bytes_written":152441,"cfile_init":1,"dirs.queue_time_us":3076,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":9999,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1457,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8358,"thread_start_us":74,"threads_started":1}
I20240430 10:33:32.489656 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 134903 bytes on disk
I20240430 10:33:32.490460 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240430 10:33:32.490995 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014414
I20240430 10:33:32.727586 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.000000
I20240430 10:33:32.789314 25274 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 851 ops behind the committed index  [suppressed 6 similar messages]
I20240430 10:33:32.919775 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:33.482717 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 10:33:33.539678 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.811s	user 0.236s	sys 0.004s Metrics: {"bytes_written":149961,"cfile_init":1,"dirs.queue_time_us":54285,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1036,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5297,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8278,"thread_start_us":70,"threads_started":1}
I20240430 10:33:33.540549 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9): perf score=1.014402
W20240430 10:33:33.874387 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240430 10:33:33.922801 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 10:33:34.372871 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.882s	user 1.402s	sys 0.012s Metrics: {"bytes_written":11141,"cfile_cache_hit":16,"cfile_cache_hit_bytes":594418,"cfile_cache_miss":146,"cfile_cache_miss_bytes":3358666,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":12343,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":4791,"drs_written":1,"lbm_read_time_us":6670,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":177,"lbm_write_time_us":7165,"lbm_writes_lt_1ms":123,"num_input_rowsets":2,"peak_mem_usage":1878903,"reinsert_count":0,"rows_written":57320,"spinlock_wait_cycles":9549184,"thread_start_us":433,"threads_started":7,"update_count":1726}
I20240430 10:33:34.375226 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 1102344 bytes on disk
I20240430 10:33:34.375797 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240430 10:33:34.378751 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(6ebcf145701c453694b3c0e260fd396b): perf score=1.000000
I20240430 10:33:34.656839 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.278s	user 0.215s	sys 0.000s Metrics: {"bytes_written":140179,"cfile_init":1,"dirs.queue_time_us":317,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1447,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6446,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":7659}
I20240430 10:33:34.657685 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(6ebcf145701c453694b3c0e260fd396b): perf score=1.014717
I20240430 10:33:34.880096 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:35.001657 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 1.461s	user 1.283s	sys 0.012s Metrics: {"bytes_written":371,"cfile_cache_hit":14,"cfile_cache_hit_bytes":413978,"cfile_cache_miss":152,"cfile_cache_miss_bytes":3626294,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":14445,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1010,"drs_written":1,"lbm_read_time_us":4937,"lbm_reads_lt_1ms":188,"lbm_write_time_us":23550,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":125,"num_input_rowsets":2,"peak_mem_usage":1936868,"reinsert_count":0,"rows_written":58141,"thread_start_us":87,"threads_started":1,"update_count":30}
I20240430 10:33:35.002310 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9): 1133948 bytes on disk
I20240430 10:33:35.002710 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240430 10:33:35.003408 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(6ebcf145701c453694b3c0e260fd396b): perf score=1.000000
I20240430 10:33:35.299417 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.642s	user 0.475s	sys 0.004s Metrics: {"bytes_written":1901,"cfile_cache_hit":13,"cfile_cache_hit_bytes":132047,"cfile_cache_miss":55,"cfile_cache_miss_bytes":926311,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":16550,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":1631,"drs_written":1,"lbm_read_time_us":1908,"lbm_reads_lt_1ms":87,"lbm_write_time_us":2558,"lbm_writes_lt_1ms":54,"mutex_wait_us":31,"num_input_rowsets":2,"peak_mem_usage":489421,"reinsert_count":0,"rows_written":15461,"spinlock_wait_cycles":7552,"thread_start_us":595,"threads_started":8,"update_count":266}
I20240430 10:33:35.300647 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b): 299787 bytes on disk
I20240430 10:33:35.301141 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240430 10:33:35.301844 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=1.000000
I20240430 10:33:35.304718 25304 consensus_queue.cc:786] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 21 ops behind the committed index 
I20240430 10:33:35.347206 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.343s	user 0.213s	sys 0.004s Metrics: {"bytes_written":139680,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":790,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1530,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7658}
I20240430 10:33:35.348227 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b): 145895 bytes on disk
I20240430 10:33:35.351184 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240430 10:33:35.351814 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(6ebcf145701c453694b3c0e260fd396b): perf score=1.014715
W20240430 10:33:35.380131 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240430 10:33:35.479291 25247 consensus_queue.cc:786] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 17 ops behind the committed index 
I20240430 10:33:35.545509 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:35.598341 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.296s	user 0.286s	sys 0.000s Metrics: {"bytes_written":140901,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1000,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1530,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7718}
I20240430 10:33:35.599506 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b): 145751 bytes on disk
I20240430 10:33:35.605274 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240430 10:33:35.606182 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling CompactRowSetsOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=1.014761
I20240430 10:33:35.669318 24421 linked_list-test-util.h:417] Sample at HT timestamp: 7022482291361853440 Inserted count: 94250
I20240430 10:33:35.669504 24421 linked_list-test-util.h:421] Finished inserting list. Added 94250 in chain
I20240430 10:33:35.669569 24421 linked_list-test-util.h:422] Last entries inserted had keys:
I20240430 10:33:35.669636 24421 linked_list-test-util.h:424] 0: -4598352857741197312
I20240430 10:33:35.669708 24421 linked_list-test-util.h:424] 1: -7868596861849894911
I20240430 10:33:35.669795 24421 linked_list-test-util.h:424] 2: 7308325420215959554
I20240430 10:33:35.669858 24421 linked_list-test-util.h:424] 3: 4038081416107261955
I20240430 10:33:35.669911 24421 linked_list-test-util.h:424] 4: -3830233970765922300
I20240430 10:33:35.669965 24421 linked_list-test-util.h:424] 5: -7100477974874619899
I20240430 10:33:35.670027 24421 linked_list-test-util.h:424] 6: 8076444307191234566
I20240430 10:33:35.670085 24421 linked_list-test-util.h:424] 7: 4806200303082536967
I20240430 10:33:35.670138 24421 linked_list-test-util.h:424] 8: -3062115083790647288
I20240430 10:33:35.670192 24421 linked_list-test-util.h:424] 9: -6332218350411055095
I20240430 10:33:35.670246 24421 linked_list-test-util.h:424] 10: 8844563194166509578
I20240430 10:33:35.670298 24421 linked_list-test-util.h:424] 11: 5574459927546101771
I20240430 10:33:35.670369 24421 linked_list-test-util.h:424] 12: -2293996196815372276
I20240430 10:33:35.670426 24421 linked_list-test-util.h:424] 13: -5564099463435780083
I20240430 10:33:35.670539 24421 linked_list-test-util.h:424] 14: -8834061992567767026
I20240430 10:33:35.670595 24421 linked_list-test-util.h:424] 15: 6342578814521376783
I20240430 10:33:35.670651 24421 linked_list-test-util.h:424] 16: -1525736572351807472
I20240430 10:33:35.670706 24421 linked_list-test-util.h:424] 17: -4795980576460505071
I20240430 10:33:35.670763 24421 linked_list-test-util.h:424] 18: -8065943105592492014
I20240430 10:33:35.670817 24421 linked_list-test-util.h:424] 19: 7110697701496651795
I20240430 10:33:35.670871 24421 linked_list-test-util.h:424] 20: -757617685376532460
I20240430 10:33:35.670946 24421 linked_list-test-util.h:424] 21: -4027861689485230059
I20240430 10:33:35.671006 24421 linked_list-test-util.h:424] 22: -7297683481128927210
I20240430 10:33:35.671062 24421 linked_list-test-util.h:424] 23: 7878816588471926807
I20240430 10:33:35.671118 24421 linked_list-test-util.h:424] 24: 10501201598742552
I20240430 10:33:35.671173 24421 linked_list-test-util.h:424] 25: -3259742802509955047
I20240430 10:33:35.671228 24421 linked_list-test-util.h:424] 26: -6529846069130362854
I20240430 10:33:35.671283 24421 linked_list-test-util.h:424] 27: 4048582617706004507
I20240430 10:33:35.671341 24421 linked_list-test-util.h:424] 28: 778620088574017564
I20240430 10:33:35.671397 24421 linked_list-test-util.h:424] 29: -2491483178046390243
I20240430 10:33:35.671463 24421 linked_list-test-util.h:424] 30: -5761727182155087842
I20240430 10:33:35.671643 24421 linked_list-test-util.h:424] 31: 4816701504681279519
I20240430 10:33:35.671707 24421 linked_list-test-util.h:424] 32: 1546738975549292576
I20240430 10:33:35.671759 24421 linked_list-test-util.h:424] 33: -1723364291071115231
I20240430 10:33:35.671811 24421 linked_list-test-util.h:424] 34: -4993608295179812830
I20240430 10:33:35.671864 24421 linked_list-test-util.h:424] 35: 5584820391656554531
I20240430 10:33:35.671914 24421 linked_list-test-util.h:424] 36: 2314857862524567588
I20240430 10:33:35.671968 24421 linked_list-test-util.h:424] 37: -955245404095840219
I20240430 10:33:35.672020 24421 linked_list-test-util.h:424] 38: -4225489408204537818
I20240430 10:33:35.672070 24421 linked_list-test-util.h:424] 39: 6352939278631829543
I20240430 10:33:35.672122 24421 linked_list-test-util.h:424] 40: 3083117486988132392
I20240430 10:33:35.672173 24421 linked_list-test-util.h:424] 41: -187126517120565207
I20240430 10:33:35.672225 24421 linked_list-test-util.h:424] 42: -3457229783740973014
I20240430 10:33:35.672276 24421 linked_list-test-util.h:424] 43: 7121058165607104555
I20240430 10:33:35.672327 24421 linked_list-test-util.h:424] 44: 3851236373963407404
I20240430 10:33:35.672377 24421 linked_list-test-util.h:424] 45: 580992369854709805
I20240430 10:33:35.672430 24421 linked_list-test-util.h:424] 46: -2689110896765698002
I20240430 10:33:35.672487 24421 linked_list-test-util.h:424] 47: 7889317790070669359
I20240430 10:33:35.672536 24421 linked_list-test-util.h:424] 48: 4619073785961971760
I20240430 10:33:35.672585 24421 linked_list-test-util.h:424] 49: 1349111256829984817
W20240430 10:33:35.742671 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240430 10:33:35.793532 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 1714473195647284 usec, L: 1 Expected Rows: 30150
I20240430 10:33:35.808566 24421 linked_list-test-util.h:528] Calling callback on tserver cf979890494e40c8b3207effd7c83cef
I20240430 10:33:35.837863 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.486s	user 0.348s	sys 0.008s Metrics: {"bytes_written":487,"cfile_cache_hit":12,"cfile_cache_hit_bytes":43926,"cfile_cache_miss":60,"cfile_cache_miss_bytes":1033622,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":851,"drs_written":1,"lbm_read_time_us":5723,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":91,"lbm_write_time_us":2616,"lbm_writes_lt_1ms":54,"mutex_wait_us":26,"num_input_rowsets":2,"peak_mem_usage":511490,"reinsert_count":0,"rows_written":15598,"spinlock_wait_cycles":17024,"thread_start_us":464,"threads_started":7,"update_count":48}
I20240430 10:33:35.838670 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=1.000000
I20240430 10:33:35.905606 25235 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 985 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:36.117614 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: FlushMRSOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.279s	user 0.208s	sys 0.000s Metrics: {"bytes_written":140319,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1006,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1607,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7728}
I20240430 10:33:36.118774 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling CompactRowSetsOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=1.014760
I20240430 10:33:36.152750 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: CompactRowSetsOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.546s	user 0.521s	sys 0.004s Metrics: {"bytes_written":1473,"cfile_cache_hit":14,"cfile_cache_hit_bytes":132993,"cfile_cache_miss":58,"cfile_cache_miss_bytes":943199,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":31706,"dirs.run_cpu_time_us":865,"dirs.run_wall_time_us":5720,"drs_written":1,"lbm_read_time_us":2231,"lbm_reads_lt_1ms":90,"lbm_write_time_us":2840,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":508640,"reinsert_count":0,"rows_written":15603,"thread_start_us":401,"threads_started":7,"update_count":201}
I20240430 10:33:36.153564 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9): perf score=0.012415
I20240430 10:33:36.260895 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(07177c6fdc5742458352ef97b02bebb9) complete. Timing: real 0.107s	user 0.104s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":16206,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":74,"lbm_read_time_us":821,"lbm_reads_lt_1ms":20,"lbm_write_time_us":243,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1726}
I20240430 10:33:36.261658 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(6ebcf145701c453694b3c0e260fd396b): perf score=0.007860
I20240430 10:33:36.290729 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.029s	user 0.028s	sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2789,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":79,"lbm_read_time_us":246,"lbm_reads_lt_1ms":20,"lbm_write_time_us":210,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":266}
I20240430 10:33:36.291471 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling MajorDeltaCompactionOp(e1b2f890834447e1b34a35e5bcf1203b): perf score=0.006039
I20240430 10:33:36.319613 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: MajorDeltaCompactionOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.028s	user 0.026s	sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2183,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":73,"lbm_read_time_us":276,"lbm_reads_lt_1ms":24,"lbm_write_time_us":216,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":201}
I20240430 10:33:36.320372 24915 maintenance_manager.cc:419] P 2dffd8c702cd4fdba865ead4911f8584: Scheduling UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b): 303202 bytes on disk
I20240430 10:33:36.320895 24805 maintenance_manager.cc:643] P 2dffd8c702cd4fdba865ead4911f8584: UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240430 10:33:36.365942 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240430 10:33:36.391479 24421 meta_cache.cc:1510] marking tablet server f2aa0f551289403caa641628dd390846 (127.23.217.66:45035) as failed
W20240430 10:33:36.391589 24421 meta_cache.cc:302] tablet 07177c6fdc5742458352ef97b02bebb9: replica f2aa0f551289403caa641628dd390846 (127.23.217.66:45035) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111)
I20240430 10:33:36.543812 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: CompactRowSetsOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.422s	user 0.399s	sys 0.004s Metrics: {"bytes_written":286492,"cfile_cache_hit":14,"cfile_cache_hit_bytes":123392,"cfile_cache_miss":54,"cfile_cache_miss_bytes":965950,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":830,"drs_written":1,"lbm_read_time_us":1726,"lbm_reads_lt_1ms":86,"lbm_write_time_us":2577,"lbm_writes_lt_1ms":50,"num_input_rowsets":2,"peak_mem_usage":514560,"rows_written":15740}
I20240430 10:33:36.544535 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b): 307383 bytes on disk
I20240430 10:33:36.545012 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: UndoDeltaBlockGCOp(e1b2f890834447e1b34a35e5bcf1203b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240430 10:33:36.545449 24653 maintenance_manager.cc:419] P cf979890494e40c8b3207effd7c83cef: Scheduling MajorDeltaCompactionOp(6ebcf145701c453694b3c0e260fd396b): perf score=0.002002
W20240430 10:33:36.584419 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240430 10:33:36.586035 24543 maintenance_manager.cc:643] P cf979890494e40c8b3207effd7c83cef: MajorDeltaCompactionOp(6ebcf145701c453694b3c0e260fd396b) complete. Timing: real 0.040s	user 0.038s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":264,"cfile_cache_miss":2,"cfile_cache_miss_bytes":386,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":57,"lbm_read_time_us":201,"lbm_reads_lt_1ms":10,"lbm_write_time_us":290,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":48}
I20240430 10:33:36.587232 24421 linked_list-test-util.h:767] Done collecting results (30150 rows in 778.777ms)
I20240430 10:33:36.623143 24421 linked_list-test-util.h:556] Snapshot: P: 1714473195647284 usec, L: 1 verified. Result: OK
I20240430 10:33:36.626026 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 1714473205649780 usec, L: 1 Expected Rows: 65450
I20240430 10:33:36.633383 24421 linked_list-test-util.h:528] Calling callback on tserver cf979890494e40c8b3207effd7c83cef
I20240430 10:33:37.049688 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:37.113881 24541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240430 10:33:37.414505 24421 linked_list-test-util.h:767] Done collecting results (65450 rows in 781.194ms)
I20240430 10:33:37.460220 24421 linked_list-test-util.h:556] Snapshot: P: 1714473205649780 usec, L: 1 verified. Result: OK
I20240430 10:33:37.462280 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 1714473215664515 usec, L: 1 Expected Rows: 94250
I20240430 10:33:37.468458 24421 linked_list-test-util.h:528] Calling callback on tserver cf979890494e40c8b3207effd7c83cef
I20240430 10:33:37.501562 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:38.453691 25362 consensus_queue.cc:786] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 39 ops behind the committed index  [suppressed 5 similar messages]
W20240430 10:33:38.472096 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240430 10:33:38.515496 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:38.610108 25317 consensus_queue.cc:786] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 26 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:38.749251 24421 linked_list-test-util.h:767] Done collecting results (94250 rows in 1280.87ms)
I20240430 10:33:38.820231 24421 linked_list-test-util.h:556] Snapshot: P: 1714473215664515 usec, L: 1 verified. Result: OK
I20240430 10:33:38.822400 24421 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 94250
I20240430 10:33:38.969791 25362 consensus_queue.cc:786] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 985 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:38.969862 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240430 10:33:38.976055 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240430 10:33:39.218397 24541 consensus_peers.cc:485] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240430 10:33:39.990773 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:40.474965 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:40.563568 24421 linked_list-test-util.h:767] Done collecting results (94250 rows in 1470.73ms)
I20240430 10:33:40.647261 24421 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240430 10:33:40.647596 24421 linked_list-test-util.h:692] Successfully verified 94250 rows
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs: 
TabletServer: f2aa0f551289403caa641628dd390846, Rpc address: host: "127.23.217.66" port: 45035 (Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111))
I20240430 10:33:40.658015 24421 webui_checker.cc:88] shutting down CURL thread
W20240430 10:33:40.797094 24803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240430 10:33:40.887014 24803 consensus_peers.cc:485] T 6ebcf145701c453694b3c0e260fd396b P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240430 10:33:41.267515 24803 consensus_peers.cc:485] T 07177c6fdc5742458352ef97b02bebb9 P 2dffd8c702cd4fdba865ead4911f8584 -> Peer f2aa0f551289403caa641628dd390846 (127.23.217.66:45035): Couldn't send request to peer f2aa0f551289403caa641628dd390846. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.66:45035: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240430 10:33:41.375916 24955 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:41.776963 24957 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240430 10:33:41.784718 25317 consensus_queue.cc:786] T e1b2f890834447e1b34a35e5bcf1203b P cf979890494e40c8b3207effd7c83cef [LEADER]: Peer f2aa0f551289403caa641628dd390846 is lagging by at least 26 ops behind the committed index  [suppressed 5 similar messages]
I20240430 10:33:41.789422 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 24524
I20240430 10:33:41.799050 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 24786
I20240430 10:33:41.808035 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 24433
2024-04-30T10:33:41Z chronyd exiting
I20240430 10:33:41.825536 24421 test_util.cc:175] -----------------------------------------------
I20240430 10:33:41.825588 24421 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714473184567055-24421-0
[  FAILED  ] LinkedListTest.TestLoadAndVerify (37249 ms)
[ RUN      ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240430 10:33:41.826202 24421 test_util.cc:268] Using random seed: -413325674
I20240430 10:33:41.826887 24421 linked_list-test.cc:100] Linked List Test Configuration:
I20240430 10:33:41.826953 24421 linked_list-test.cc:101] --------------
I20240430 10:33:41.826974 24421 linked_list-test.cc:102] 50 chains
I20240430 10:33:41.826992 24421 linked_list-test.cc:103] 3 tablets
I20240430 10:33:41.827018 24421 linked_list-test.cc:104] Mutations on
I20240430 10:33:41.827041 24421 linked_list-test.cc:105] --------------
I20240430 10:33:41.827097 24421 ts_itest-base.cc:112] Starting cluster with:
I20240430 10:33:41.827121 24421 ts_itest-base.cc:113] --------------
I20240430 10:33:41.827140 24421 ts_itest-base.cc:114] 3 tablet servers
I20240430 10:33:41.827163 24421 ts_itest-base.cc:115] 3 replicas per TS
I20240430 10:33:41.827188 24421 ts_itest-base.cc:116] --------------
2024-04-30T10:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T10:33:41Z Disabled control of system clock
I20240430 10:33:41.835315 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.217.126:42027
--webserver_interface=127.23.217.126
--webserver_port=0
--builtin_ntp_servers=127.23.217.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.217.126:42027
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240430 10:33:41.942078 25397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:41.942322 25397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:41.942396 25397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:41.945401 25397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 10:33:41.945464 25397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:41.945497 25397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 10:33:41.945525 25397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 10:33:41.949172 25397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.217.126:42027
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.217.126:42027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.23.217.126
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.25397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:41.950095 25397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:41.951257 25397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:41.959008 25402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:41.959623 25403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:41.960204 25405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:41.960827 25397 server_base.cc:1034] running on GCE node
I20240430 10:33:41.961336 25397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:41.962283 25397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:41.963446 25397 hybrid_clock.cc:648] HybridClock initialized: now 1714473221963423 us; error 42 us; skew 500 ppm
I20240430 10:33:41.965409 25397 webserver.cc:460] Webserver started at http://127.23.217.126:40663/ using document root <none> and password file <none>
I20240430 10:33:41.965975 25397 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:41.966045 25397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:41.966215 25397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:41.967845 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/instance:
uuid: "873dbf66bdf14944a42787e791f6b34e"
format_stamp: "Formatted at 2024-04-30 10:33:41 on dist-test-slave-k5tk"
I20240430 10:33:41.968307 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal/instance:
uuid: "873dbf66bdf14944a42787e791f6b34e"
format_stamp: "Formatted at 2024-04-30 10:33:41 on dist-test-slave-k5tk"
I20240430 10:33:41.971521 25397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240430 10:33:41.973590 25411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:41.974525 25397 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240430 10:33:41.974635 25397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
uuid: "873dbf66bdf14944a42787e791f6b34e"
format_stamp: "Formatted at 2024-04-30 10:33:41 on dist-test-slave-k5tk"
I20240430 10:33:41.974714 25397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:41.987380 25397 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:41.988107 25397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:41.988322 25397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:41.994526 25397 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.126:42027
I20240430 10:33:41.994531 25463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.126:42027 every 8 connection(s)
I20240430 10:33:41.995530 25397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/data/info.pb
I20240430 10:33:41.997963 25464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:42.000283 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 25397
I20240430 10:33:42.000435 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/master-0/wal/instance
I20240430 10:33:42.003509 25464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e: Bootstrap starting.
I20240430 10:33:42.005487 25464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:42.006171 25464 log.cc:824] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:42.007903 25464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e: No bootstrap required, opened a new log
I20240430 10:33:42.010499 25464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873dbf66bdf14944a42787e791f6b34e" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 42027 } }
I20240430 10:33:42.010665 25464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:42.010700 25464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873dbf66bdf14944a42787e791f6b34e, State: Initialized, Role: FOLLOWER
I20240430 10:33:42.011085 25464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873dbf66bdf14944a42787e791f6b34e" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 42027 } }
I20240430 10:33:42.011210 25464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 10:33:42.011252 25464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 10:33:42.011345 25464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:42.012009 25464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873dbf66bdf14944a42787e791f6b34e" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 42027 } }
I20240430 10:33:42.012336 25464 leader_election.cc:302] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 873dbf66bdf14944a42787e791f6b34e; no voters: 
I20240430 10:33:42.012632 25464 leader_election.cc:288] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240430 10:33:42.012710 25469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:33:42.012930 25469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [term 1 LEADER]: Becoming Leader. State: Replica: 873dbf66bdf14944a42787e791f6b34e, State: Running, Role: LEADER
I20240430 10:33:42.013221 25469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873dbf66bdf14944a42787e791f6b34e" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 42027 } }
I20240430 10:33:42.013676 25464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [sys.catalog]: configured and running, proceeding with master startup.
I20240430 10:33:42.014976 25471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 873dbf66bdf14944a42787e791f6b34e. Latest consensus state: current_term: 1 leader_uuid: "873dbf66bdf14944a42787e791f6b34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873dbf66bdf14944a42787e791f6b34e" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 42027 } } }
I20240430 10:33:42.015087 25471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [sys.catalog]: This master's current role is: LEADER
I20240430 10:33:42.014995 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "873dbf66bdf14944a42787e791f6b34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "873dbf66bdf14944a42787e791f6b34e" member_type: VOTER last_known_addr { host: "127.23.217.126" port: 42027 } } }
I20240430 10:33:42.015304 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e [sys.catalog]: This master's current role is: LEADER
I20240430 10:33:42.015957 25473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 10:33:42.018675 25473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 10:33:42.024574 25473 catalog_manager.cc:1347] Generated new cluster ID: b35fb1675d66439a9b1208f2877820b4
I20240430 10:33:42.024633 25473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 10:33:42.034530 25473 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 10:33:42.035398 25473 catalog_manager.cc:1504] Loading token signing keys...
I20240430 10:33:42.044389 25473 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 873dbf66bdf14944a42787e791f6b34e: Generated new TSK 0
I20240430 10:33:42.044999 25473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 10:33:42.053694 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.65:0
--local_ip_for_outbound_sockets=127.23.217.65
--webserver_interface=127.23.217.65
--webserver_port=0
--tserver_master_addrs=127.23.217.126:42027
--builtin_ntp_servers=127.23.217.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 10:33:42.159709 25488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:42.159911 25488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:42.159963 25488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:42.163008 25488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:42.163122 25488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.65
I20240430 10:33:42.166658 25488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.23.217.65
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.23.217.126:42027
--never_fsync=true
--heap_profile_path=/tmp/kudu.25488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.65
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:42.167541 25488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:42.168677 25488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:42.176995 25493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:42.177595 25496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:42.178238 25494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:42.178923 25488 server_base.cc:1034] running on GCE node
I20240430 10:33:42.179291 25488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:42.179838 25488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:42.180994 25488 hybrid_clock.cc:648] HybridClock initialized: now 1714473222180966 us; error 39 us; skew 500 ppm
I20240430 10:33:42.182878 25488 webserver.cc:460] Webserver started at http://127.23.217.65:41143/ using document root <none> and password file <none>
I20240430 10:33:42.183467 25488 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:42.183526 25488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:42.183709 25488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:42.185302 25488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/instance:
uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.185758 25488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal/instance:
uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.188921 25488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.003s
I20240430 10:33:42.190979 25502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.191929 25488 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:33:42.192042 25488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.192135 25488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:42.208170 25488 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:42.208806 25488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:42.208967 25488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:42.209513 25488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:33:42.210393 25488 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:33:42.210443 25488 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.210502 25488 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:33:42.210534 25488 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.218808 25488 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.65:35615
I20240430 10:33:42.218851 25615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.65:35615 every 8 connection(s)
I20240430 10:33:42.219743 25488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
I20240430 10:33:42.227957 25616 heartbeater.cc:347] Connected to a master server at 127.23.217.126:42027
I20240430 10:33:42.228169 25616 heartbeater.cc:464] Registering TS with master...
I20240430 10:33:42.228668 25616 heartbeater.cc:511] Master 127.23.217.126:42027 requested a full tablet report, sending...
I20240430 10:33:42.229555 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 25488
I20240430 10:33:42.229671 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal/instance
I20240430 10:33:42.230167 25428 ts_manager.cc:194] Registered new tserver with Master: 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615)
I20240430 10:33:42.231503 25428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.65:45905
I20240430 10:33:42.231915 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.66:0
--local_ip_for_outbound_sockets=127.23.217.66
--webserver_interface=127.23.217.66
--webserver_port=0
--tserver_master_addrs=127.23.217.126:42027
--builtin_ntp_servers=127.23.217.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 10:33:42.335224 25619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:42.335433 25619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:42.335481 25619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:42.338438 25619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:42.338548 25619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.66
I20240430 10:33:42.342010 25619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.23.217.66
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.23.217.126:42027
--never_fsync=true
--heap_profile_path=/tmp/kudu.25619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.66
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:42.342875 25619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:42.343971 25619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:42.352190 25627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:42.352718 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:42.353346 25625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:42.354032 25619 server_base.cc:1034] running on GCE node
I20240430 10:33:42.354393 25619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:42.354950 25619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:42.356109 25619 hybrid_clock.cc:648] HybridClock initialized: now 1714473222356087 us; error 38 us; skew 500 ppm
I20240430 10:33:42.357892 25619 webserver.cc:460] Webserver started at http://127.23.217.66:40669/ using document root <none> and password file <none>
I20240430 10:33:42.358430 25619 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:42.358503 25619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:42.358666 25619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:42.360318 25619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/instance:
uuid: "467752939d6b40bea56af2111e42cd81"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.360778 25619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal/instance:
uuid: "467752939d6b40bea56af2111e42cd81"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.363843 25619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240430 10:33:42.365893 25633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.366827 25619 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240430 10:33:42.366953 25619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
uuid: "467752939d6b40bea56af2111e42cd81"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.367044 25619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:42.390566 25619 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:42.391294 25619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:42.391453 25619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:42.391970 25619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:33:42.392813 25619 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:33:42.392863 25619 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.392938 25619 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:33:42.392976 25619 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.401790 25619 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.66:39069
I20240430 10:33:42.401831 25746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.66:39069 every 8 connection(s)
I20240430 10:33:42.402688 25619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/data/info.pb
I20240430 10:33:42.407780 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 25619
I20240430 10:33:42.407907 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-1/wal/instance
I20240430 10:33:42.410102 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.67:0
--local_ip_for_outbound_sockets=127.23.217.67
--webserver_interface=127.23.217.67
--webserver_port=0
--tserver_master_addrs=127.23.217.126:42027
--builtin_ntp_servers=127.23.217.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240430 10:33:42.411412 25747 heartbeater.cc:347] Connected to a master server at 127.23.217.126:42027
I20240430 10:33:42.411640 25747 heartbeater.cc:464] Registering TS with master...
I20240430 10:33:42.412107 25747 heartbeater.cc:511] Master 127.23.217.126:42027 requested a full tablet report, sending...
I20240430 10:33:42.413053 25428 ts_manager.cc:194] Registered new tserver with Master: 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069)
I20240430 10:33:42.413575 25428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.66:58629
W20240430 10:33:42.515326 25750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:33:42.515549 25750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:33:42.515619 25750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:33:42.518580 25750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:33:42.518714 25750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.67
I20240430 10:33:42.522257 25750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.23.217.67
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.23.217.126:42027
--never_fsync=true
--heap_profile_path=/tmp/kudu.25750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.67
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:33:42.523151 25750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:33:42.524277 25750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:33:42.533058 25758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:42.533565 25756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:33:42.534221 25755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:33:42.534956 25750 server_base.cc:1034] running on GCE node
I20240430 10:33:42.535329 25750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:33:42.535846 25750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:33:42.536994 25750 hybrid_clock.cc:648] HybridClock initialized: now 1714473222536986 us; error 34 us; skew 500 ppm
I20240430 10:33:42.538806 25750 webserver.cc:460] Webserver started at http://127.23.217.67:35447/ using document root <none> and password file <none>
I20240430 10:33:42.539382 25750 fs_manager.cc:359] Metadata directory not provided
I20240430 10:33:42.539444 25750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:33:42.539619 25750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 10:33:42.541234 25750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/instance:
uuid: "1c7a296e22494559aa5cc142bef58e9a"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.541699 25750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal/instance:
uuid: "1c7a296e22494559aa5cc142bef58e9a"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.545061 25750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.001s
I20240430 10:33:42.547195 25764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.548151 25750 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240430 10:33:42.548264 25750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
uuid: "1c7a296e22494559aa5cc142bef58e9a"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:33:42.548348 25750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:33:42.559110 25750 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:33:42.559700 25750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:33:42.559849 25750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:33:42.560361 25750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:33:42.561288 25750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 10:33:42.561343 25750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.561439 25750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 10:33:42.561475 25750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:33:42.569931 25750 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.67:38813
I20240430 10:33:42.569957 25877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.67:38813 every 8 connection(s)
I20240430 10:33:42.570878 25750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/data/info.pb
I20240430 10:33:42.575840 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 25750
I20240430 10:33:42.575951 24421 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-2/wal/instance
I20240430 10:33:42.579473 25878 heartbeater.cc:347] Connected to a master server at 127.23.217.126:42027
I20240430 10:33:42.579672 25878 heartbeater.cc:464] Registering TS with master...
I20240430 10:33:42.580155 25878 heartbeater.cc:511] Master 127.23.217.126:42027 requested a full tablet report, sending...
I20240430 10:33:42.581041 25428 ts_manager.cc:194] Registered new tserver with Master: 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813)
I20240430 10:33:42.581544 25428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.67:46743
I20240430 10:33:42.589848 24421 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 10:33:42.598233 25428 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43586:
name: "TestTable"
schema {
  columns {
    name: "rand_key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "link_to"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "insert_ts"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "updated"
    type: BOOL
    is_key: false
    is_nullable: false
    read_default_value: "\000"
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "rand_key"
    }
  }
}
W20240430 10:33:42.600958 25428 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 10:33:42.616595 25812 tablet_service.cc:1467] Processing CreateTablet for tablet 36685ac17d6540beadcfce790bca32f1 (DEFAULT_TABLE table=TestTable [id=bd12cc1495ce4ee59edcd20f6b127e76]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 10:33:42.616595 25550 tablet_service.cc:1467] Processing CreateTablet for tablet 36685ac17d6540beadcfce790bca32f1 (DEFAULT_TABLE table=TestTable [id=bd12cc1495ce4ee59edcd20f6b127e76]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 10:33:42.616595 25681 tablet_service.cc:1467] Processing CreateTablet for tablet 36685ac17d6540beadcfce790bca32f1 (DEFAULT_TABLE table=TestTable [id=bd12cc1495ce4ee59edcd20f6b127e76]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240430 10:33:42.617487 25812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36685ac17d6540beadcfce790bca32f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:42.617504 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36685ac17d6540beadcfce790bca32f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:42.617511 25681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36685ac17d6540beadcfce790bca32f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 10:33:42.622586 25895 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a: Bootstrap starting.
I20240430 10:33:42.623858 25896 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: Bootstrap starting.
I20240430 10:33:42.623876 25897 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Bootstrap starting.
I20240430 10:33:42.624898 25895 tablet_bootstrap.cc:654] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:42.625536 25895 log.cc:824] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:42.625798 25897 tablet_bootstrap.cc:654] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:42.625798 25896 tablet_bootstrap.cc:654] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: Neither blocks nor log segments found. Creating new log.
I20240430 10:33:42.626621 25896 log.cc:824] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:42.626621 25897 log.cc:824] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Log is configured to *not* fsync() on all Append() calls
I20240430 10:33:42.627295 25895 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a: No bootstrap required, opened a new log
I20240430 10:33:42.627473 25895 ts_tablet_manager.cc:1367] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240430 10:33:42.628173 25896 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: No bootstrap required, opened a new log
I20240430 10:33:42.628324 25896 ts_tablet_manager.cc:1367] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.001s
I20240430 10:33:42.630597 25895 raft_consensus.cc:348] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.630793 25895 raft_consensus.cc:374] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:42.630853 25895 raft_consensus.cc:729] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7a296e22494559aa5cc142bef58e9a, State: Initialized, Role: FOLLOWER
I20240430 10:33:42.631271 25895 consensus_queue.cc:260] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.631500 25896 raft_consensus.cc:348] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.631670 25896 raft_consensus.cc:374] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:42.631734 25896 raft_consensus.cc:729] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467752939d6b40bea56af2111e42cd81, State: Initialized, Role: FOLLOWER
I20240430 10:33:42.631927 25878 heartbeater.cc:503] Master 127.23.217.126:42027 was elected leader, sending a full tablet report...
I20240430 10:33:42.632095 25895 ts_tablet_manager.cc:1398] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.003s
I20240430 10:33:42.632145 25896 consensus_queue.cc:260] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.632735 25747 heartbeater.cc:503] Master 127.23.217.126:42027 was elected leader, sending a full tablet report...
I20240430 10:33:42.632946 25896 ts_tablet_manager.cc:1398] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.005s
I20240430 10:33:42.634187 25897 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: No bootstrap required, opened a new log
I20240430 10:33:42.634390 25897 ts_tablet_manager.cc:1367] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.003s
I20240430 10:33:42.637979 25897 raft_consensus.cc:348] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.638154 25897 raft_consensus.cc:374] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 10:33:42.638185 25897 raft_consensus.cc:729] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fb2a4d507344dfbccb38ef5bc2bb9b, State: Initialized, Role: FOLLOWER
I20240430 10:33:42.638494 25897 consensus_queue.cc:260] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.639070 25616 heartbeater.cc:503] Master 127.23.217.126:42027 was elected leader, sending a full tablet report...
I20240430 10:33:42.639205 25897 ts_tablet_manager.cc:1398] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.001s
W20240430 10:33:42.653537 25748 tablet.cc:2369] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:33:42.694746 25901 raft_consensus.cc:482] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 10:33:42.694984 25901 raft_consensus.cc:504] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.696094 25901 leader_election.cc:288] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615), 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069)
I20240430 10:33:42.700122 25570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685ac17d6540beadcfce790bca32f1" candidate_uuid: "1c7a296e22494559aa5cc142bef58e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" is_pre_election: true
I20240430 10:33:42.700453 25570 raft_consensus.cc:2454] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7a296e22494559aa5cc142bef58e9a in term 0.
I20240430 10:33:42.700448 25701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685ac17d6540beadcfce790bca32f1" candidate_uuid: "1c7a296e22494559aa5cc142bef58e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467752939d6b40bea56af2111e42cd81" is_pre_election: true
I20240430 10:33:42.700699 25701 raft_consensus.cc:2454] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7a296e22494559aa5cc142bef58e9a in term 0.
I20240430 10:33:42.700939 25767 leader_election.cc:302] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7a296e22494559aa5cc142bef58e9a, 92fb2a4d507344dfbccb38ef5bc2bb9b; no voters: 
I20240430 10:33:42.701224 25901 raft_consensus.cc:2781] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 10:33:42.701290 25901 raft_consensus.cc:482] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 10:33:42.701323 25901 raft_consensus.cc:3037] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:42.702203 25901 raft_consensus.cc:504] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.702526 25901 leader_election.cc:288] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [CANDIDATE]: Term 1 election: Requested vote from peers 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615), 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069)
I20240430 10:33:42.702842 25570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685ac17d6540beadcfce790bca32f1" candidate_uuid: "1c7a296e22494559aa5cc142bef58e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b"
I20240430 10:33:42.702932 25701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685ac17d6540beadcfce790bca32f1" candidate_uuid: "1c7a296e22494559aa5cc142bef58e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467752939d6b40bea56af2111e42cd81"
I20240430 10:33:42.702968 25570 raft_consensus.cc:3037] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:42.703068 25701 raft_consensus.cc:3037] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 0 FOLLOWER]: Advancing to term 1
I20240430 10:33:42.703820 25570 raft_consensus.cc:2454] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7a296e22494559aa5cc142bef58e9a in term 1.
I20240430 10:33:42.703872 25701 raft_consensus.cc:2454] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7a296e22494559aa5cc142bef58e9a in term 1.
I20240430 10:33:42.704154 25767 leader_election.cc:302] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7a296e22494559aa5cc142bef58e9a, 467752939d6b40bea56af2111e42cd81; no voters: 
I20240430 10:33:42.704335 25901 raft_consensus.cc:2781] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 10:33:42.704569 25901 raft_consensus.cc:686] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [term 1 LEADER]: Becoming Leader. State: Replica: 1c7a296e22494559aa5cc142bef58e9a, State: Running, Role: LEADER
I20240430 10:33:42.704890 25901 consensus_queue.cc:237] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:33:42.707948 25428 catalog_manager.cc:5484] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67). New cstate: current_term: 1 leader_uuid: "1c7a296e22494559aa5cc142bef58e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } health_report { overall_health: UNKNOWN } } }
W20240430 10:33:42.720791 25617 tablet.cc:2369] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 10:33:42.730788 24421 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 10:33:42.732146 24421 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 92fb2a4d507344dfbccb38ef5bc2bb9b to finish bootstrapping
I20240430 10:33:42.737380 24421 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 467752939d6b40bea56af2111e42cd81 to finish bootstrapping
I20240430 10:33:42.742374 24421 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1c7a296e22494559aa5cc142bef58e9a to finish bootstrapping
I20240430 10:33:42.746872 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 25488
I20240430 10:33:42.755720 24421 linked_list-test-util.h:401] Running for: 30.000s
I20240430 10:33:42.755776 24421 linked_list-test-util.h:402] Sampling every 10000000 us
I20240430 10:33:42.755796 24421 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
   0%  (min) = 0
  25%        = 0
  50%  (med) = 0
  75%        = 0
  95%        = 0
  99%        = 0
  99.9%      = 0
  99.99%     = 0
  100% (max) = 0
W20240430 10:33:42.768426 25767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111)
I20240430 10:33:42.768728 25701 raft_consensus.cc:1261] T 36685ac17d6540beadcfce790bca32f1 P 467752939d6b40bea56af2111e42cd81 [term 1 FOLLOWER]: Refusing update from remote peer 1c7a296e22494559aa5cc142bef58e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240430 10:33:42.769335 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 10:33:42.769348 25901 consensus_queue.cc:1035] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 10:33:45.201995 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 10:33:46.203176 25901 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 50 ops behind the committed index 
W20240430 10:33:47.683898 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 10:33:48.187969 25767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240430 10:33:49.509593 25901 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 572 ops behind the committed index  [suppressed 6 similar messages]
W20240430 10:33:50.133682 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240430 10:33:52.635646 25953 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 1047 ops behind the committed index  [suppressed 5 similar messages]
W20240430 10:33:52.641456 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240430 10:33:52.756554 24421 linked_list-test-util.h:417] Sample at HT timestamp: 7022482361354416128 Inserted count: 79150
W20240430 10:33:53.614197 25767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240430 10:33:55.330965 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240430 10:33:55.825599 25953 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 1537 ops behind the committed index  [suppressed 5 similar messages]
W20240430 10:33:57.936297 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240430 10:33:58.732581 25767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240430 10:33:59.329669 25901 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 2063 ops behind the committed index  [suppressed 6 similar messages]
W20240430 10:34:00.236060 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240430 10:34:02.526504 25901 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 2551 ops behind the committed index  [suppressed 6 similar messages]
W20240430 10:34:02.531881 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240430 10:34:02.757229 24421 linked_list-test-util.h:417] Sample at HT timestamp: 7022482402318897152 Inserted count: 154700
W20240430 10:34:03.945299 25767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240430 10:34:05.004396 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240430 10:34:05.973709 25981 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 3085 ops behind the committed index  [suppressed 6 similar messages]
W20240430 10:34:07.524506 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240430 10:34:09.133299 26056 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 3576 ops behind the committed index  [suppressed 5 similar messages]
W20240430 10:34:09.138116 25767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240430 10:34:10.308240 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240430 10:34:12.141775 26097 consensus_queue.cc:786] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a [LEADER]: Peer 92fb2a4d507344dfbccb38ef5bc2bb9b is lagging by at least 4030 ops behind the committed index  [suppressed 5 similar messages]
W20240430 10:34:12.639647 25767 consensus_peers.cc:485] T 36685ac17d6540beadcfce790bca32f1 P 1c7a296e22494559aa5cc142bef58e9a -> Peer 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615): Couldn't send request to peer 92fb2a4d507344dfbccb38ef5bc2bb9b. Status: Network error: Client connection negotiation failed: client connection to 127.23.217.65:35615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240430 10:34:12.757048 24421 linked_list-test-util.h:417] Sample at HT timestamp: 7022482443278721024 Inserted count: 230550
I20240430 10:34:12.757314 24421 linked_list-test-util.h:421] Finished inserting list. Added 230550 in chain
I20240430 10:34:12.757398 24421 linked_list-test-util.h:422] Last entries inserted had keys:
I20240430 10:34:12.757457 24421 linked_list-test-util.h:424] 0: 8596029223054737408
I20240430 10:34:12.757525 24421 linked_list-test-util.h:424] 1: -6329458578097897471
I20240430 10:34:12.757599 24421 linked_list-test-util.h:424] 2: -2807920830564270078
I20240430 10:34:12.757632 24421 linked_list-test-util.h:424] 3: 713335441992646659
I20240430 10:34:12.757647 24421 linked_list-test-util.h:424] 4: -5615982398616829948
I20240430 10:34:12.757661 24421 linked_list-test-util.h:424] 5: -2094444651083202555
I20240430 10:34:12.757675 24421 linked_list-test-util.h:424] 6: 1426811621473714182
I20240430 10:34:12.757689 24421 linked_list-test-util.h:424] 7: 4948349369007341575
I20240430 10:34:12.757725 24421 linked_list-test-util.h:424] 8: -1380827734113845240
I20240430 10:34:12.757755 24421 linked_list-test-util.h:424] 9: 2140428538443071497
I20240430 10:34:12.757792 24421 linked_list-test-util.h:424] 10: 5661966285976698890
I20240430 10:34:12.757833 24421 linked_list-test-util.h:424] 11: 9183504033510326283
I20240430 10:34:12.757858 24421 linked_list-test-util.h:424] 12: 2853904717924139020
I20240430 10:34:12.757874 24421 linked_list-test-util.h:424] 13: 6375442465457766413
I20240430 10:34:12.757907 24421 linked_list-test-util.h:424] 14: -8549763860718157810
I20240430 10:34:12.757931 24421 linked_list-test-util.h:424] 15: -5028507588161241073
I20240430 10:34:12.757946 24421 linked_list-test-util.h:424] 16: 7089059382427123728
I20240430 10:34:12.757985 24421 linked_list-test-util.h:424] 17: -7836146943748800495
I20240430 10:34:12.758010 24421 linked_list-test-util.h:424] 18: -4314890671191883758
I20240430 10:34:12.758025 24421 linked_list-test-util.h:424] 19: -793352923658256365
I20240430 10:34:12.758065 24421 linked_list-test-util.h:424] 20: -7122670764267732972
I20240430 10:34:12.758090 24421 linked_list-test-util.h:424] 21: -3601414491710816235
I20240430 10:34:12.758133 24421 linked_list-test-util.h:424] 22: -79876744177188842
I20240430 10:34:12.758198 24421 linked_list-test-util.h:424] 23: 3441379528379727895
I20240430 10:34:12.758224 24421 linked_list-test-util.h:424] 24: -2887797574741458920
I20240430 10:34:12.758239 24421 linked_list-test-util.h:424] 25: 633740172792168473
I20240430 10:34:12.758280 24421 linked_list-test-util.h:424] 26: 4154855707860795418
I20240430 10:34:12.758304 24421 linked_list-test-util.h:424] 27: -2174321395260391397
I20240430 10:34:12.758319 24421 linked_list-test-util.h:424] 28: 1347216352273235996
I20240430 10:34:12.758354 24421 linked_list-test-util.h:424] 29: 4868472624830152733
I20240430 10:34:12.758397 24421 linked_list-test-util.h:424] 30: 8390010372363780126
I20240430 10:34:12.758412 24421 linked_list-test-util.h:424] 31: 2060692531754303519
I20240430 10:34:12.758453 24421 linked_list-test-util.h:424] 32: 5581948804311220256
I20240430 10:34:12.758479 24421 linked_list-test-util.h:424] 33: 9103486551844847649
I20240430 10:34:12.758495 24421 linked_list-test-util.h:424] 34: -5821719774331076574
I20240430 10:34:12.758533 24421 linked_list-test-util.h:424] 35: 6295565721280577571
I20240430 10:34:12.758558 24421 linked_list-test-util.h:424] 36: -8629640604895346652
I20240430 10:34:12.758574 24421 linked_list-test-util.h:424] 37: -5108102857361719259
I20240430 10:34:12.758615 24421 linked_list-test-util.h:424] 38: -1586846584804802522
I20240430 10:34:12.758641 24421 linked_list-test-util.h:424] 39: -7916164425414279129
I20240430 10:34:12.758656 24421 linked_list-test-util.h:424] 40: -4394908152857362392
I20240430 10:34:12.758693 24421 linked_list-test-util.h:424] 41: -873370405323734999
I20240430 10:34:12.758718 24421 linked_list-test-util.h:424] 42: 2648167342209892394
I20240430 10:34:12.758733 24421 linked_list-test-util.h:424] 43: -3681291235888005077
I20240430 10:34:12.758774 24421 linked_list-test-util.h:424] 44: -159753488354377684
I20240430 10:34:12.758800 24421 linked_list-test-util.h:424] 45: 3361784259179249709
I20240430 10:34:12.758816 24421 linked_list-test-util.h:424] 46: 6883040531736166446
I20240430 10:34:12.758860 24421 linked_list-test-util.h:424] 47: 553722691126689839
I20240430 10:34:12.758886 24421 linked_list-test-util.h:424] 48: 4075260438660317232
I20240430 10:34:12.758921 24421 linked_list-test-util.h:424] 49: 7596516711217233969
I20240430 10:34:12.791690 24421 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
/tmp/dist-test-task_Uyv1E/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.217.65:35615
--local_ip_for_outbound_sockets=127.23.217.65
--tserver_master_addrs=127.23.217.126:42027
--webserver_port=41143
--webserver_interface=127.23.217.65
--builtin_ntp_servers=127.23.217.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240430 10:34:12.902295 26108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 10:34:12.902523 26108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 10:34:12.902602 26108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 10:34:12.905625 26108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 10:34:12.905799 26108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.217.65
I20240430 10:34:12.909399 26108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.217.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.217.65:35615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.23.217.65
--webserver_port=41143
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.23.217.126:42027
--never_fsync=true
--heap_profile_path=/tmp/kudu.26108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.217.65
--log_dir=/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type DEBUG
built by None at 30 Apr 2024 10:13:57 UTC on 1a15035a6830
build id 53
I20240430 10:34:12.910318 26108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 10:34:12.911484 26108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 10:34:12.919657 26116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:34:12.920256 26114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 10:34:12.920864 26113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 10:34:12.921579 26108 server_base.cc:1034] running on GCE node
I20240430 10:34:12.921980 26108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 10:34:12.922592 26108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 10:34:12.923748 26108 hybrid_clock.cc:648] HybridClock initialized: now 1714473252923737 us; error 24 us; skew 500 ppm
I20240430 10:34:12.925712 26108 webserver.cc:460] Webserver started at http://127.23.217.65:41143/ using document root <none> and password file <none>
I20240430 10:34:12.926266 26108 fs_manager.cc:359] Metadata directory not provided
I20240430 10:34:12.926342 26108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 10:34:12.929656 26108 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:34:12.931596 26122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240430 10:34:12.932623 26108 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240430 10:34:12.932744 26108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data,/tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b"
format_stamp: "Formatted at 2024-04-30 10:33:42 on dist-test-slave-k5tk"
I20240430 10:34:12.933176 26108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 10:34:12.948920 26108 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240430 10:34:12.949604 26108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 10:34:12.949775 26108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 10:34:12.950332 26108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 10:34:12.951501 26129 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240430 10:34:12.954787 26108 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240430 10:34:12.954860 26108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20240430 10:34:12.955003 26108 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240430 10:34:12.956555 26108 ts_tablet_manager.cc:590] Registered 1 tablets
I20240430 10:34:12.956606 26108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20240430 10:34:12.956940 26129 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Bootstrap starting.
I20240430 10:34:12.967447 26108 rpc_server.cc:307] RPC server started. Bound to: 127.23.217.65:35615
I20240430 10:34:12.967499 26236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.217.65:35615 every 8 connection(s)
I20240430 10:34:12.968547 26108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/data/info.pb
I20240430 10:34:12.969260 24421 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu as pid 26108
I20240430 10:34:12.970456 26129 log.cc:824] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Log is configured to *not* fsync() on all Append() calls
I20240430 10:34:12.972079 26129 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 10:34:12.972354 26129 log.cc:1225] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-task_Uyv1E/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714473184567055-24421-0/linked-list-cluster/ts-0/wal/wals/36685ac17d6540beadcfce790bca32f1.recovery-1714473252972298
I20240430 10:34:12.972481 26129 tablet_bootstrap.cc:492] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Bootstrap complete.
I20240430 10:34:12.972783 26129 ts_tablet_manager.cc:1367] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.006s
I20240430 10:34:12.976861 26129 raft_consensus.cc:348] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:34:12.977594 26129 raft_consensus.cc:729] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fb2a4d507344dfbccb38ef5bc2bb9b, State: Initialized, Role: FOLLOWER
I20240430 10:34:12.978155 26129 consensus_queue.cc:260] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:34:12.978228 26237 heartbeater.cc:347] Connected to a master server at 127.23.217.126:42027
I20240430 10:34:12.978475 26237 heartbeater.cc:464] Registering TS with master...
I20240430 10:34:12.979117 26129 ts_tablet_manager.cc:1398] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.005s
I20240430 10:34:12.979100 26237 heartbeater.cc:511] Master 127.23.217.126:42027 requested a full tablet report, sending...
I20240430 10:34:12.981043 25428 ts_manager.cc:194] Re-registered known tserver with Master: 92fb2a4d507344dfbccb38ef5bc2bb9b (127.23.217.65:35615)
I20240430 10:34:12.981820 25428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.217.65:43507
I20240430 10:34:12.983656 26237 heartbeater.cc:503] Master 127.23.217.126:42027 was elected leader, sending a full tablet report...
I20240430 10:34:12.987462 24421 cluster_itest_util.cc:256] Not converged past 4611 yet: 1.4642 1.4642 0.0
I20240430 10:34:13.111150 24421 cluster_itest_util.cc:256] Not converged past 4611 yet: 1.4642 1.4642 1.525
I20240430 10:34:13.313786 24421 cluster_itest_util.cc:256] Not converged past 4611 yet: 1.4642 1.4642 1.1001
I20240430 10:34:13.489495 26247 mvcc.cc:205] Tried to move back new op lower bound from 7022482333098975232 to 7022482320201867264. Current Snapshot: MvccSnapshot[applied={T|T < 7022482329044983808 or (T in {7022482329108725760,7022482329137184768,7022482329166450688,7022482329048862720,7022482329219059712,7022482329193537536,7022482329296924672,7022482329324916736,7022482329351168000,7022482329245769728,7022482329436266496,7022482329459236864,7022482329482678272,7022482329380798464,7022482329532833792,7022482329600032768,7022482329557938176,7022482329624666112,7022482329510060032,7022482329649278976,7022482329698439168,7022482329721401344,7022482329745375232,7022482329673932800,7022482329769766912,7022482329789964288,7022482329813716992,7022482329883058176,7022482329904652288,7022482329861943296,7022482329837957120,7022482329948270592,7022482329970020352,7022482330017624064,7022482330042376192,7022482330064232448,7022482330086481920,7022482329926828032,7022482329995124736,7022482330156314624,7022482330132631552,7022482330203115520,7022482330178224128,7022482330108563456,7022482330225721344,7022482330246881280,7022482330270982144,7022482330292649984,7022482330320371712,7022482330390700032,7022482330435256320,7022482330458632192,7022482330484084736,7022482330506379264,7022482330528538624,7022482330554044416,7022482330576756736,7022482330601402368,7022482330358120448,7022482330668789760,7022482330691723264,7022482330713501696,7022482330738835456,7022482330760749056,7022482330784440320,7022482330809663488,7022482330831282176,7022482330645962752,7022482330874929152,7022482330899816448,7022482330623442944,7022482330924302336,7022482330947047424,7022482330969161728,7022482330993532928,7022482331017666560,7022482331061420032,7022482331039084544,7022482331108220928,7022482330852880384,7022482331133145088,7022482331084193792,7022482331179040768,7022482331155222528,7022482331246718976,7022482331224748032,7022482331293294592,7022482331201150976,7022482331339866112,7022482331362140160,7022482331383955456,7022482331428691968,7022482331315449856,7022482331451777024,7022482331406888960,7022482331473657856,7022482331497213952,7022482331521339392,7022482331569467392,7022482331592957952,7022482331616571392,7022482331641126912,7022482331689160704,7022482331713236992,7022482331545980928,7022482331761627136,7022482331786854400,7022482331809050624,7022482331665174528,7022482331854569472,7022482331880361984,7022482331901710336})}]
I20240430 10:34:13.616302 24421 cluster_itest_util.cc:256] Not converged past 4611 yet: 1.4642 1.4642 1.1970
I20240430 10:34:14.019345 24421 cluster_itest_util.cc:256] Not converged past 4611 yet: 1.4642 1.4642 1.3429
I20240430 10:34:14.521788 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 25619
I20240430 10:34:14.532680 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 25750
I20240430 10:34:14.547273 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 1714473232752543 usec, L: 1 Expected Rows: 79150
I20240430 10:34:14.553764 24421 meta_cache.cc:1510] marking tablet server 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) as failed
W20240430 10:34:14.553860 24421 meta_cache.cc:302] tablet 36685ac17d6540beadcfce790bca32f1: replica 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
I20240430 10:34:14.599242 24421 linked_list-test-util.h:528] Calling callback on tserver 92fb2a4d507344dfbccb38ef5bc2bb9b
I20240430 10:34:15.812322 26258 raft_consensus.cc:482] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c7a296e22494559aa5cc142bef58e9a)
I20240430 10:34:15.812633 26258 raft_consensus.cc:504] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:34:15.814313 26258 leader_election.cc:288] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813), 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069)
W20240430 10:34:15.814711 26123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
W20240430 10:34:15.815830 26123 leader_election.cc:334] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813): Network error: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
W20240430 10:34:15.816641 26125 leader_election.cc:334] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069): Network error: Client connection negotiation failed: client connection to 127.23.217.66:39069: connect: Connection refused (error 111)
I20240430 10:34:15.816712 26125 leader_election.cc:302] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fb2a4d507344dfbccb38ef5bc2bb9b; no voters: 1c7a296e22494559aa5cc142bef58e9a, 467752939d6b40bea56af2111e42cd81
I20240430 10:34:15.816952 26258 raft_consensus.cc:2726] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:34:16.070709 24421 linked_list-test-util.h:767] Done collecting results (79150 rows in 1471.63ms)
I20240430 10:34:16.193043 24421 linked_list-test-util.h:556] Snapshot: P: 1714473232752543 usec, L: 1 verified. Result: OK
I20240430 10:34:16.195355 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 1714473242753637 usec, L: 1 Expected Rows: 154700
I20240430 10:34:16.196538 24421 meta_cache.cc:1510] marking tablet server 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) as failed
W20240430 10:34:16.196599 24421 meta_cache.cc:302] tablet 36685ac17d6540beadcfce790bca32f1: replica 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
I20240430 10:34:16.219049 24421 linked_list-test-util.h:528] Calling callback on tserver 92fb2a4d507344dfbccb38ef5bc2bb9b
I20240430 10:34:17.348024 26263 raft_consensus.cc:482] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c7a296e22494559aa5cc142bef58e9a)
I20240430 10:34:17.348608 26263 raft_consensus.cc:504] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:34:17.349473 26263 leader_election.cc:288] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813), 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069)
W20240430 10:34:17.365550 26123 leader_election.cc:334] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813): Network error: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
W20240430 10:34:17.365664 26125 leader_election.cc:334] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069): Network error: Client connection negotiation failed: client connection to 127.23.217.66:39069: connect: Connection refused (error 111)
I20240430 10:34:17.365727 26125 leader_election.cc:302] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fb2a4d507344dfbccb38ef5bc2bb9b; no voters: 1c7a296e22494559aa5cc142bef58e9a, 467752939d6b40bea56af2111e42cd81
I20240430 10:34:17.365886 26263 raft_consensus.cc:2726] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:34:18.600714 24421 linked_list-test-util.h:767] Done collecting results (154700 rows in 2381.75ms)
I20240430 10:34:18.716974 24421 linked_list-test-util.h:556] Snapshot: P: 1714473242753637 usec, L: 1 verified. Result: OK
I20240430 10:34:18.719096 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 1714473252753594 usec, L: 1 Expected Rows: 230550
I20240430 10:34:18.720054 24421 meta_cache.cc:1510] marking tablet server 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) as failed
W20240430 10:34:18.720121 24421 meta_cache.cc:302] tablet 36685ac17d6540beadcfce790bca32f1: replica 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
I20240430 10:34:18.726356 24421 linked_list-test-util.h:528] Calling callback on tserver 92fb2a4d507344dfbccb38ef5bc2bb9b
I20240430 10:34:19.429869 26268 raft_consensus.cc:482] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c7a296e22494559aa5cc142bef58e9a)
I20240430 10:34:19.429996 26268 raft_consensus.cc:504] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92fb2a4d507344dfbccb38ef5bc2bb9b" member_type: VOTER last_known_addr { host: "127.23.217.65" port: 35615 } } peers { permanent_uuid: "1c7a296e22494559aa5cc142bef58e9a" member_type: VOTER last_known_addr { host: "127.23.217.67" port: 38813 } } peers { permanent_uuid: "467752939d6b40bea56af2111e42cd81" member_type: VOTER last_known_addr { host: "127.23.217.66" port: 39069 } }
I20240430 10:34:19.430428 26268 leader_election.cc:288] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813), 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069)
W20240430 10:34:19.431640 26125 leader_election.cc:334] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 467752939d6b40bea56af2111e42cd81 (127.23.217.66:39069): Network error: Client connection negotiation failed: client connection to 127.23.217.66:39069: connect: Connection refused (error 111)
W20240430 10:34:19.431862 26123 leader_election.cc:334] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813): Network error: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
I20240430 10:34:19.431946 26123 leader_election.cc:302] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fb2a4d507344dfbccb38ef5bc2bb9b; no voters: 1c7a296e22494559aa5cc142bef58e9a, 467752939d6b40bea56af2111e42cd81
I20240430 10:34:19.432081 26268 raft_consensus.cc:2726] T 36685ac17d6540beadcfce790bca32f1 P 92fb2a4d507344dfbccb38ef5bc2bb9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 10:34:20.766448 24421 linked_list-test-util.h:767] Done collecting results (230550 rows in 2040.17ms)
I20240430 10:34:20.943040 24421 linked_list-test-util.h:556] Snapshot: P: 1714473252753594 usec, L: 1 verified. Result: OK
I20240430 10:34:20.945235 24421 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 230550
I20240430 10:34:20.946211 24421 meta_cache.cc:1510] marking tablet server 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) as failed
W20240430 10:34:20.946297 24421 meta_cache.cc:302] tablet 36685ac17d6540beadcfce790bca32f1: replica 1c7a296e22494559aa5cc142bef58e9a (127.23.217.67:38813) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.217.67:38813: connect: Connection refused (error 111)
I20240430 10:34:20.949373 24421 linked_list-test-util.h:528] Calling callback on tserver 92fb2a4d507344dfbccb38ef5bc2bb9b
I20240430 10:34:22.960529 24421 linked_list-test-util.h:767] Done collecting results (230550 rows in 2011.23ms)
I20240430 10:34:23.139851 24421 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240430 10:34:23.140153 24421 linked_list-test-util.h:692] Successfully verified 230550 rows
W20240430 10:34:23.140720 25883 client.cc:2025] Couldn't close scanner 96dd1857550d41ef8d41e39b42e8959b: Service unavailable: reactor is shutting down (error 108)
I20240430 10:34:23.140726 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 26108
I20240430 10:34:23.151300 24421 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Uyv1E/build/debug/bin/kudu with pid 25397
2024-04-30T10:34:23Z chronyd exiting
[       OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (41351 ms)
[----------] 2 tests from LinkedListTest (78600 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (78601 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LinkedListTest.TestLoadAndVerify

 1 FAILED TEST