[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LinkedListTest
[ RUN ] LinkedListTest.TestLoadAndVerify
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 20:45:24.772599 8724 test_util.cc:268] Using random seed: -1363776826
I20240502 20:45:24.779104 8724 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 20:45:24.779150 8724 linked_list-test.cc:101] --------------
I20240502 20:45:24.779165 8724 linked_list-test.cc:102] 50 chains
I20240502 20:45:24.779176 8724 linked_list-test.cc:103] 3 tablets
I20240502 20:45:24.779188 8724 linked_list-test.cc:104] Mutations on
I20240502 20:45:24.779201 8724 linked_list-test.cc:105] --------------
I20240502 20:45:24.779275 8724 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:24.779300 8724 ts_itest-base.cc:113] --------------
I20240502 20:45:24.779314 8724 ts_itest-base.cc:114] 3 tablet servers
I20240502 20:45:24.779330 8724 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:24.779351 8724 ts_itest-base.cc:116] --------------
2024-05-02T20:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:24Z Disabled control of system clock
I20240502 20:45:24.789326 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.133.62:42121
--webserver_interface=127.8.133.62
--webserver_port=0
--builtin_ntp_servers=127.8.133.20:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.133.62:42121
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 20:45:24.892521 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:24.892742 8736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:24.892786 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:24.895785 8736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:24.895848 8736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:24.895866 8736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:24.895886 8736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:24.899546 8736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.133.62:42121
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.133.62:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.8.133.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.8736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:24.900408 8736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:24.901620 8736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:24.908766 8744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.909401 8742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:24.909603 8741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:24.910426 8736 server_base.cc:1034] running on GCE node
I20240502 20:45:24.910897 8736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:24.911788 8736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:24.912938 8736 hybrid_clock.cc:648] HybridClock initialized: now 1714682724912924 us; error 25 us; skew 500 ppm
I20240502 20:45:24.914700 8736 webserver.cc:460] Webserver started at http://127.8.133.62:45781/ using document root <none> and password file <none>
I20240502 20:45:24.915251 8736 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:24.915309 8736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:24.915493 8736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:24.917201 8736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/instance:
uuid: "759eb050061f4c5682e9909a7640adbe"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-3019"
I20240502 20:45:24.917663 8736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal/instance:
uuid: "759eb050061f4c5682e9909a7640adbe"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-3019"
I20240502 20:45:24.920868 8736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 20:45:24.923010 8750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:24.923957 8736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:24.924064 8736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
uuid: "759eb050061f4c5682e9909a7640adbe"
format_stamp: "Formatted at 2024-05-02 20:45:24 on dist-test-slave-3019"
I20240502 20:45:24.924145 8736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:24.938480 8736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:24.939163 8736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:24.939323 8736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:24.946195 8736 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.62:42121
I20240502 20:45:24.946249 8802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.62:42121 every 8 connection(s)
I20240502 20:45:24.947190 8736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/info.pb
I20240502 20:45:24.949965 8803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:24.954265 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 8736
I20240502 20:45:24.954402 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal/instance
I20240502 20:45:24.955459 8803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe: Bootstrap starting.
I20240502 20:45:24.958251 8803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:24.959173 8803 log.cc:824] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:24.960925 8803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe: No bootstrap required, opened a new log
I20240502 20:45:24.963562 8803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759eb050061f4c5682e9909a7640adbe" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 42121 } }
I20240502 20:45:24.963752 8803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:24.963835 8803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759eb050061f4c5682e9909a7640adbe, State: Initialized, Role: FOLLOWER
I20240502 20:45:24.964262 8803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759eb050061f4c5682e9909a7640adbe" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 42121 } }
I20240502 20:45:24.964392 8803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:24.964452 8803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:24.964553 8803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:24.965364 8803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759eb050061f4c5682e9909a7640adbe" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 42121 } }
I20240502 20:45:24.965695 8803 leader_election.cc:302] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759eb050061f4c5682e9909a7640adbe; no voters:
I20240502 20:45:24.965986 8803 leader_election.cc:288] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:45:24.966361 8808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:24.966615 8808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [term 1 LEADER]: Becoming Leader. State: Replica: 759eb050061f4c5682e9909a7640adbe, State: Running, Role: LEADER
I20240502 20:45:24.966990 8808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759eb050061f4c5682e9909a7640adbe" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 42121 } }
I20240502 20:45:24.967278 8803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:24.968971 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759eb050061f4c5682e9909a7640adbe. Latest consensus state: current_term: 1 leader_uuid: "759eb050061f4c5682e9909a7640adbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759eb050061f4c5682e9909a7640adbe" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 42121 } } }
I20240502 20:45:24.969156 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:24.969482 8809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759eb050061f4c5682e9909a7640adbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759eb050061f4c5682e9909a7640adbe" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 42121 } } }
I20240502 20:45:24.969897 8809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe [sys.catalog]: This master's current role is: LEADER
W20240502 20:45:24.972178 8822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 20:45:24.972239 8822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 20:45:24.972348 8823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:24.973341 8823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:24.978431 8823 catalog_manager.cc:1347] Generated new cluster ID: ded5133788fc42b89638ed44eb718cd7
I20240502 20:45:24.978505 8823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:24.997200 8823 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:24.998152 8823 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:25.003281 8823 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 759eb050061f4c5682e9909a7640adbe: Generated new TSK 0
I20240502 20:45:25.003981 8823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:25.007579 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.1:0
--local_ip_for_outbound_sockets=127.8.133.1
--webserver_interface=127.8.133.1
--webserver_port=0
--tserver_master_addrs=127.8.133.62:42121
--builtin_ntp_servers=127.8.133.20:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
W20240502 20:45:25.134184 8827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.134414 8827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.134480 8827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.137594 8827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.137666 8827 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 20:45:25.137681 8827 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 20:45:25.137703 8827 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 20:45:25.137766 8827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.1
I20240502 20:45:25.141438 8827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.8.133.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.8.133.62:42121
--never_fsync=true
--heap_profile_path=/tmp/kudu.8827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.1
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.142374 8827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.143509 8827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.151324 8833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.151580 8832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.152284 8835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.153076 8827 server_base.cc:1034] running on GCE node
I20240502 20:45:25.153409 8827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.153919 8827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.155059 8827 hybrid_clock.cc:648] HybridClock initialized: now 1714682725155044 us; error 32 us; skew 500 ppm
I20240502 20:45:25.156805 8827 webserver.cc:460] Webserver started at http://127.8.133.1:45557/ using document root <none> and password file <none>
I20240502 20:45:25.157339 8827 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.157395 8827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.157562 8827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.159137 8827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/instance:
uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.159576 8827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal/instance:
uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.162693 8827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:25.164758 8841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.165720 8827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:25.165835 8827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.165915 8827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.184055 8827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.184741 8827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.184897 8827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.185453 8827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.186357 8827 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.186401 8827 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.186455 8827 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.186480 8827 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.196359 8827 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.1:46619
I20240502 20:45:25.196400 8954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.1:46619 every 8 connection(s)
I20240502 20:45:25.197283 8827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
I20240502 20:45:25.203739 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 8827
I20240502 20:45:25.203857 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal/instance
I20240502 20:45:25.206198 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.2:0
--local_ip_for_outbound_sockets=127.8.133.2
--webserver_interface=127.8.133.2
--webserver_port=0
--tserver_master_addrs=127.8.133.62:42121
--builtin_ntp_servers=127.8.133.20:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 20:45:25.206331 8955 heartbeater.cc:347] Connected to a master server at 127.8.133.62:42121
I20240502 20:45:25.206532 8955 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.207021 8955 heartbeater.cc:511] Master 127.8.133.62:42121 requested a full tablet report, sending...
I20240502 20:45:25.208613 8767 ts_manager.cc:194] Registered new tserver with Master: 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619)
I20240502 20:45:25.209749 8767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.1:60189
W20240502 20:45:25.311275 8958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.311467 8958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.311543 8958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.314478 8958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.314550 8958 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 20:45:25.314574 8958 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 20:45:25.314608 8958 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 20:45:25.314671 8958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.2
I20240502 20:45:25.318152 8958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.8.133.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.8.133.62:42121
--never_fsync=true
--heap_profile_path=/tmp/kudu.8958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.2
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.318965 8958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.320055 8958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.327770 8964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.328519 8966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.328734 8963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.329576 8958 server_base.cc:1034] running on GCE node
I20240502 20:45:25.329946 8958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.330508 8958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.331669 8958 hybrid_clock.cc:648] HybridClock initialized: now 1714682725331646 us; error 33 us; skew 500 ppm
I20240502 20:45:25.333549 8958 webserver.cc:460] Webserver started at http://127.8.133.2:34327/ using document root <none> and password file <none>
I20240502 20:45:25.334163 8958 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.334225 8958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.334398 8958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.335958 8958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/instance:
uuid: "023da3a68b244e89b00248344f835b3a"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.336423 8958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal/instance:
uuid: "023da3a68b244e89b00248344f835b3a"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.339563 8958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 20:45:25.341675 8972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.342643 8958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:25.342746 8958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
uuid: "023da3a68b244e89b00248344f835b3a"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.342821 8958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.379835 8958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.380607 8958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.380753 8958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.381342 8958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.382359 8958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.382409 8958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.382462 8958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.382486 8958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.392294 8958 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.2:36163
I20240502 20:45:25.392385 9085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.2:36163 every 8 connection(s)
I20240502 20:45:25.393296 8958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/info.pb
I20240502 20:45:25.401943 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 8958
I20240502 20:45:25.402067 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal/instance
I20240502 20:45:25.402101 9086 heartbeater.cc:347] Connected to a master server at 127.8.133.62:42121
I20240502 20:45:25.402330 9086 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.402902 9086 heartbeater.cc:511] Master 127.8.133.62:42121 requested a full tablet report, sending...
I20240502 20:45:25.403889 8767 ts_manager.cc:194] Registered new tserver with Master: 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163)
I20240502 20:45:25.404286 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.3:0
--local_ip_for_outbound_sockets=127.8.133.3
--webserver_interface=127.8.133.3
--webserver_port=0
--tserver_master_addrs=127.8.133.62:42121
--builtin_ntp_servers=127.8.133.20:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tablet_compaction_budget_mb=4
--tablet_delta_store_major_compact_min_ratio=0.001
--log_segment_size_mb=1 with env {}
I20240502 20:45:25.404525 8767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.2:36979
W20240502 20:45:25.515224 9089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:25.515396 9089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:25.515451 9089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:25.518381 9089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:25.518445 9089 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20240502 20:45:25.518462 9089 flags.cc:425] Enabled experimental flag: --tablet_compaction_budget_mb=4
W20240502 20:45:25.518500 9089 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20240502 20:45:25.518564 9089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.3
I20240502 20:45:25.522063 9089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.8.133.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=4
--tablet_history_max_age_sec=86400
--flush_threshold_mb=1
--tserver_master_addrs=127.8.133.62:42121
--never_fsync=true
--heap_profile_path=/tmp/kudu.9089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.3
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:25.522809 9089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:25.523835 9089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:25.531898 9094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.532191 9097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:25.532318 9095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:25.532922 9089 server_base.cc:1034] running on GCE node
I20240502 20:45:25.533262 9089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:25.533753 9089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:25.534896 9089 hybrid_clock.cc:648] HybridClock initialized: now 1714682725534885 us; error 26 us; skew 500 ppm
I20240502 20:45:25.536540 9089 webserver.cc:460] Webserver started at http://127.8.133.3:45489/ using document root <none> and password file <none>
I20240502 20:45:25.537079 9089 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:25.537137 9089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:25.537298 9089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:25.538808 9089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/instance:
uuid: "5da62881d7084cbf848e7b61285e41a0"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.539250 9089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal/instance:
uuid: "5da62881d7084cbf848e7b61285e41a0"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.542304 9089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:45:25.544294 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.545475 9089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:45:25.545589 9089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
uuid: "5da62881d7084cbf848e7b61285e41a0"
format_stamp: "Formatted at 2024-05-02 20:45:25 on dist-test-slave-3019"
I20240502 20:45:25.545678 9089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:25.556924 9089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:45:25.557526 9089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:25.557670 9089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:25.558184 9089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:25.559056 9089 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:25.559101 9089 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.559151 9089 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:25.559180 9089 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:45:25.568866 9089 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.3:42481
I20240502 20:45:25.568892 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.3:42481 every 8 connection(s)
I20240502 20:45:25.569820 9089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/info.pb
I20240502 20:45:25.578404 9217 heartbeater.cc:347] Connected to a master server at 127.8.133.62:42121
I20240502 20:45:25.578627 9217 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:25.579144 9217 heartbeater.cc:511] Master 127.8.133.62:42121 requested a full tablet report, sending...
I20240502 20:45:25.579912 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 9089
I20240502 20:45:25.580020 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal/instance
I20240502 20:45:25.580209 8767 ts_manager.cc:194] Registered new tserver with Master: 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
I20240502 20:45:25.580859 8767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.3:56931
I20240502 20:45:25.582665 8724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 20:45:25.593552 8767 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44180:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 20:45:25.596264 8767 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 20:45:25.612329 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 68374b60234f4f30ac844bc877056d6f (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 20:45:25.612682 9019 tablet_service.cc:1467] Processing CreateTablet for tablet 273f27ac228d4473ba17b83f75a27ea2 (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 20:45:25.613610 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68374b60234f4f30ac844bc877056d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.613684 9019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273f27ac228d4473ba17b83f75a27ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.614373 9018 tablet_service.cc:1467] Processing CreateTablet for tablet 68374b60234f4f30ac844bc877056d6f (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 20:45:25.613958 9151 tablet_service.cc:1467] Processing CreateTablet for tablet 2567a0ea66a441138ca0d35f9c477955 (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 20:45:25.614624 9018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68374b60234f4f30ac844bc877056d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.614948 9150 tablet_service.cc:1467] Processing CreateTablet for tablet 273f27ac228d4473ba17b83f75a27ea2 (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 20:45:25.615031 9151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2567a0ea66a441138ca0d35f9c477955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.615325 9020 tablet_service.cc:1467] Processing CreateTablet for tablet 2567a0ea66a441138ca0d35f9c477955 (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 20:45:25.615561 9020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2567a0ea66a441138ca0d35f9c477955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.616088 8888 tablet_service.cc:1467] Processing CreateTablet for tablet 273f27ac228d4473ba17b83f75a27ea2 (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204
I20240502 20:45:25.616366 8888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273f27ac228d4473ba17b83f75a27ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.613965 9149 tablet_service.cc:1467] Processing CreateTablet for tablet 68374b60234f4f30ac844bc877056d6f (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION 6148914691236517204 <= VALUES
I20240502 20:45:25.617724 9150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273f27ac228d4473ba17b83f75a27ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.617887 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68374b60234f4f30ac844bc877056d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.621624 9236 tablet_bootstrap.cc:492] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0: Bootstrap starting.
I20240502 20:45:25.621662 9234 tablet_bootstrap.cc:492] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: Bootstrap starting.
I20240502 20:45:25.612331 8889 tablet_service.cc:1467] Processing CreateTablet for tablet 2567a0ea66a441138ca0d35f9c477955 (DEFAULT_TABLE table=TestTable [id=c17705e7239346e5abfd7b6b952a58a5]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602
I20240502 20:45:25.622509 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2567a0ea66a441138ca0d35f9c477955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:25.623447 9236 tablet_bootstrap.cc:654] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.623719 9234 tablet_bootstrap.cc:654] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.624315 9236 log.cc:824] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:25.624840 9234 log.cc:824] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:25.625207 9235 tablet_bootstrap.cc:492] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: Bootstrap starting.
I20240502 20:45:25.626080 9236 tablet_bootstrap.cc:492] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0: No bootstrap required, opened a new log
I20240502 20:45:25.626260 9236 ts_tablet_manager.cc:1367] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:25.626971 9235 tablet_bootstrap.cc:654] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.627784 9235 log.cc:824] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:25.627965 9234 tablet_bootstrap.cc:492] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: No bootstrap required, opened a new log
I20240502 20:45:25.628158 9234 ts_tablet_manager.cc:1367] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 20:45:25.629155 9235 tablet_bootstrap.cc:492] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: No bootstrap required, opened a new log
I20240502 20:45:25.629302 9235 ts_tablet_manager.cc:1367] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 20:45:25.629462 9236 raft_consensus.cc:348] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.629664 9236 raft_consensus.cc:374] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.629719 9236 raft_consensus.cc:729] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da62881d7084cbf848e7b61285e41a0, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.630182 9236 consensus_queue.cc:260] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.630995 9236 ts_tablet_manager.cc:1398] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:45:25.631198 9234 raft_consensus.cc:348] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.631348 9217 heartbeater.cc:503] Master 127.8.133.62:42121 was elected leader, sending a full tablet report...
I20240502 20:45:25.631384 9234 raft_consensus.cc:374] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.631474 9234 raft_consensus.cc:729] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3ebe1106e84d3f91c76a45bad4bd24, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.631815 9235 raft_consensus.cc:348] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.631867 9234 consensus_queue.cc:260] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.631964 9235 raft_consensus.cc:374] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.632033 9235 raft_consensus.cc:729] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023da3a68b244e89b00248344f835b3a, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.632392 9235 consensus_queue.cc:260] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.632488 9236 tablet_bootstrap.cc:492] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0: Bootstrap starting.
I20240502 20:45:25.632645 9234 ts_tablet_manager.cc:1398] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 20:45:25.633023 9234 tablet_bootstrap.cc:492] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24: Bootstrap starting.
I20240502 20:45:25.633404 9236 tablet_bootstrap.cc:654] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.633684 9235 ts_tablet_manager.cc:1398] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 20:45:25.633852 8955 heartbeater.cc:503] Master 127.8.133.62:42121 was elected leader, sending a full tablet report...
I20240502 20:45:25.634048 9234 tablet_bootstrap.cc:654] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.634128 9086 heartbeater.cc:503] Master 127.8.133.62:42121 was elected leader, sending a full tablet report...
I20240502 20:45:25.634451 9236 tablet_bootstrap.cc:492] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0: No bootstrap required, opened a new log
I20240502 20:45:25.634524 9236 ts_tablet_manager.cc:1367] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 20:45:25.635010 9235 tablet_bootstrap.cc:492] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: Bootstrap starting.
I20240502 20:45:25.634966 9236 raft_consensus.cc:348] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.635056 9236 raft_consensus.cc:374] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.635113 9236 raft_consensus.cc:729] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da62881d7084cbf848e7b61285e41a0, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.635233 9236 consensus_queue.cc:260] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.635574 9236 ts_tablet_manager.cc:1398] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:25.635728 9236 tablet_bootstrap.cc:492] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: Bootstrap starting.
I20240502 20:45:25.636047 9235 tablet_bootstrap.cc:654] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.636493 9236 tablet_bootstrap.cc:654] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.637301 9235 tablet_bootstrap.cc:492] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: No bootstrap required, opened a new log
I20240502 20:45:25.637434 9235 ts_tablet_manager.cc:1367] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 20:45:25.637984 9235 raft_consensus.cc:348] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.638271 9236 tablet_bootstrap.cc:492] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: No bootstrap required, opened a new log
I20240502 20:45:25.638346 9236 ts_tablet_manager.cc:1367] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 20:45:25.638639 9235 raft_consensus.cc:374] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.638707 9236 raft_consensus.cc:348] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.638753 9235 raft_consensus.cc:729] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023da3a68b244e89b00248344f835b3a, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.638809 9236 raft_consensus.cc:374] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.638855 9236 raft_consensus.cc:729] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da62881d7084cbf848e7b61285e41a0, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.638972 9236 consensus_queue.cc:260] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.639032 9235 consensus_queue.cc:260] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.639212 9236 ts_tablet_manager.cc:1398] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:25.639309 9235 ts_tablet_manager.cc:1398] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 20:45:25.639556 9235 tablet_bootstrap.cc:492] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a: Bootstrap starting.
I20240502 20:45:25.639942 9234 tablet_bootstrap.cc:492] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24: No bootstrap required, opened a new log
I20240502 20:45:25.640035 9234 ts_tablet_manager.cc:1367] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 20:45:25.640602 9234 raft_consensus.cc:348] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.640708 9234 raft_consensus.cc:374] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.640746 9234 raft_consensus.cc:729] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3ebe1106e84d3f91c76a45bad4bd24, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.640873 9234 consensus_queue.cc:260] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.640941 9235 tablet_bootstrap.cc:654] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.641247 9234 ts_tablet_manager.cc:1398] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 20:45:25.641408 9234 tablet_bootstrap.cc:492] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: Bootstrap starting.
I20240502 20:45:25.642340 9235 tablet_bootstrap.cc:492] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a: No bootstrap required, opened a new log
I20240502 20:45:25.642436 9235 ts_tablet_manager.cc:1367] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 20:45:25.642530 9234 tablet_bootstrap.cc:654] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:25.642850 9235 raft_consensus.cc:348] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.642953 9235 raft_consensus.cc:374] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.643033 9235 raft_consensus.cc:729] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023da3a68b244e89b00248344f835b3a, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.643198 9235 consensus_queue.cc:260] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.643468 9235 ts_tablet_manager.cc:1398] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 20:45:25.644119 9087 tablet.cc:2369] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:25.644191 9234 tablet_bootstrap.cc:492] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: No bootstrap required, opened a new log
I20240502 20:45:25.644261 9234 ts_tablet_manager.cc:1367] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 20:45:25.644713 9234 raft_consensus.cc:348] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.644819 9234 raft_consensus.cc:374] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:25.644860 9234 raft_consensus.cc:729] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3ebe1106e84d3f91c76a45bad4bd24, State: Initialized, Role: FOLLOWER
I20240502 20:45:25.644970 9234 consensus_queue.cc:260] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.645228 9234 ts_tablet_manager.cc:1398] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 20:45:25.698383 8956 tablet.cc:2369] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:25.720652 9240 raft_consensus.cc:482] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:25.720808 9245 raft_consensus.cc:482] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:25.720870 9240 raft_consensus.cc:504] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.720927 9245 raft_consensus.cc:504] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.722019 9240 leader_election.cc:288] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163), 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619)
I20240502 20:45:25.722003 9245 leader_election.cc:288] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619), 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163)
I20240502 20:45:25.725869 8908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" is_pre_election: true
I20240502 20:45:25.725878 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a" is_pre_election: true
I20240502 20:45:25.725885 8909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" is_pre_election: true
I20240502 20:45:25.726214 8908 raft_consensus.cc:2454] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 0.
I20240502 20:45:25.726190 9040 raft_consensus.cc:2454] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 0.
I20240502 20:45:25.725878 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a" is_pre_election: true
I20240502 20:45:25.726418 9039 raft_consensus.cc:2454] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 0.
I20240502 20:45:25.726217 8909 raft_consensus.cc:2454] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 0.
I20240502 20:45:25.726734 9104 leader_election.cc:302] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 5da62881d7084cbf848e7b61285e41a0; no voters:
I20240502 20:45:25.726995 9245 raft_consensus.cc:2781] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:25.727022 9106 leader_election.cc:302] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da62881d7084cbf848e7b61285e41a0, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters:
I20240502 20:45:25.727077 9245 raft_consensus.cc:482] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:25.727123 9245 raft_consensus.cc:3037] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.728212 9245 raft_consensus.cc:504] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.728365 9240 raft_consensus.cc:2781] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:25.728425 9240 raft_consensus.cc:482] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:25.728469 9240 raft_consensus.cc:3037] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.728729 9245 leader_election.cc:288] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163), 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619)
I20240502 20:45:25.728951 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a"
I20240502 20:45:25.729054 9039 raft_consensus.cc:3037] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.729117 8909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
I20240502 20:45:25.729233 8909 raft_consensus.cc:3037] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.729449 9240 raft_consensus.cc:504] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.729805 9240 leader_election.cc:288] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619), 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163)
I20240502 20:45:25.730111 8908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
I20240502 20:45:25.730212 8908 raft_consensus.cc:3037] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.730273 9039 raft_consensus.cc:2454] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 1.
I20240502 20:45:25.730572 9104 leader_election.cc:302] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 5da62881d7084cbf848e7b61285e41a0; no voters:
I20240502 20:45:25.730743 9240 raft_consensus.cc:2781] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:25.730788 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "5da62881d7084cbf848e7b61285e41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a"
I20240502 20:45:25.730837 9240 raft_consensus.cc:686] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5da62881d7084cbf848e7b61285e41a0, State: Running, Role: LEADER
I20240502 20:45:25.730882 9039 raft_consensus.cc:3037] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.730885 8909 raft_consensus.cc:2454] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 1.
I20240502 20:45:25.731137 9240 consensus_queue.cc:237] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:25.731276 8908 raft_consensus.cc:2454] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 1.
I20240502 20:45:25.731628 9106 leader_election.cc:302] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da62881d7084cbf848e7b61285e41a0, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters:
I20240502 20:45:25.731647 9039 raft_consensus.cc:2454] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da62881d7084cbf848e7b61285e41a0 in term 1.
I20240502 20:45:25.732020 9252 raft_consensus.cc:2781] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:25.732278 9252 raft_consensus.cc:686] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5da62881d7084cbf848e7b61285e41a0, State: Running, Role: LEADER
I20240502 20:45:25.732421 9252 consensus_queue.cc:237] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.734154 8767 catalog_manager.cc:5484] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3). New cstate: current_term: 1 leader_uuid: "5da62881d7084cbf848e7b61285e41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:25.737584 8767 catalog_manager.cc:5484] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3). New cstate: current_term: 1 leader_uuid: "5da62881d7084cbf848e7b61285e41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:25.820818 9218 tablet.cc:2369] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:25.835217 9248 raft_consensus.cc:482] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:25.835454 9248 raft_consensus.cc:504] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.836663 9248 leader_election.cc:288] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163), 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
I20240502 20:45:25.840714 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273f27ac228d4473ba17b83f75a27ea2" candidate_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a" is_pre_election: true
I20240502 20:45:25.840874 9039 raft_consensus.cc:2454] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c3ebe1106e84d3f91c76a45bad4bd24 in term 0.
I20240502 20:45:25.841272 8842 leader_election.cc:302] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters:
I20240502 20:45:25.841331 9171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273f27ac228d4473ba17b83f75a27ea2" candidate_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da62881d7084cbf848e7b61285e41a0" is_pre_election: true
I20240502 20:45:25.841543 9248 raft_consensus.cc:2781] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:25.841536 9171 raft_consensus.cc:2454] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c3ebe1106e84d3f91c76a45bad4bd24 in term 0.
I20240502 20:45:25.841605 9248 raft_consensus.cc:482] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:25.841650 9248 raft_consensus.cc:3037] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.847326 9248 raft_consensus.cc:504] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.847752 9248 leader_election.cc:288] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 1 election: Requested vote from peers 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163), 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
I20240502 20:45:25.848160 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273f27ac228d4473ba17b83f75a27ea2" candidate_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a"
I20240502 20:45:25.848282 9039 raft_consensus.cc:3037] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.848413 9171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273f27ac228d4473ba17b83f75a27ea2" candidate_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da62881d7084cbf848e7b61285e41a0"
I20240502 20:45:25.848536 9171 raft_consensus.cc:3037] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:25.849370 9039 raft_consensus.cc:2454] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3ebe1106e84d3f91c76a45bad4bd24 in term 1.
I20240502 20:45:25.849515 9171 raft_consensus.cc:2454] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3ebe1106e84d3f91c76a45bad4bd24 in term 1.
I20240502 20:45:25.849728 8842 leader_election.cc:302] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters:
I20240502 20:45:25.849934 9248 raft_consensus.cc:2781] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:25.850044 9248 raft_consensus.cc:686] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3ebe1106e84d3f91c76a45bad4bd24, State: Running, Role: LEADER
I20240502 20:45:25.850370 9248 consensus_queue.cc:237] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:25.853627 8767 catalog_manager.cc:5484] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1). New cstate: current_term: 1 leader_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:25.905753 8724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 20:45:25.907555 8724 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 6c3ebe1106e84d3f91c76a45bad4bd24 to finish bootstrapping
I20240502 20:45:25.913960 8724 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 023da3a68b244e89b00248344f835b3a to finish bootstrapping
I20240502 20:45:25.918622 8724 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 5da62881d7084cbf848e7b61285e41a0 to finish bootstrapping
I20240502 20:45:25.923434 9260 webui_checker.cc:102] curl thread will poll the following URLs every 1.000s:
http://127.8.133.62:45781/ipki-ca-cert
http://127.8.133.62:45781/ipki-ca-cert-pem
http://127.8.133.3:45489/startup
http://127.8.133.2:34327/threadz
http://127.8.133.1:45557/startup
http://127.8.133.3:45489/transactions?tablet_id=68374b60234f4f30ac844bc877056d6f
http://127.8.133.62:45781/metrics
http://127.8.133.62:45781/mem-trackers
http://127.8.133.3:45489/threadz
http://127.8.133.3:45489/maintenance-manager
http://127.8.133.2:34327/maintenance-manager
http://127.8.133.2:34327/scans
http://127.8.133.3:45489/stacks
http://127.8.133.2:34327/stacks
http://127.8.133.62:45781/startup
http://127.8.133.1:45557/stacks
http://127.8.133.2:34327/threadz?group=all
http://127.8.133.3:45489/metrics
http://127.8.133.62:45781/masters
http://127.8.133.62:45781/tablet-servers
http://127.8.133.2:34327/transactions?tablet_id=68374b60234f4f30ac844bc877056d6f
http://127.8.133.62:45781/stacks
http://127.8.133.1:45557/maintenance-manager
http://127.8.133.2:34327/startup
http://127.8.133.2:34327/tablets
http://127.8.133.1:45557/threadz?group=all
http://127.8.133.2:34327/mem-trackers
http://127.8.133.3:45489/scans
http://127.8.133.1:45557/threadz
http://127.8.133.62:45781/tables
http://127.8.133.1:45557/metrics
http://127.8.133.1:45557/tablets
http://127.8.133.62:45781/threadz
http://127.8.133.1:45557/transactions?tablet_id=68374b60234f4f30ac844bc877056d6f
http://127.8.133.62:45781/threadz?group=all
http://127.8.133.3:45489/threadz?group=all
http://127.8.133.1:45557/scans
http://127.8.133.62:45781/ipki-ca-cert-der
http://127.8.133.2:34327/metrics
http://127.8.133.1:45557/mem-trackers
http://127.8.133.3:45489/tablets
http://127.8.133.62:45781/dump-entities
http://127.8.133.3:45489/mem-trackers
I20240502 20:45:25.926462 8724 linked_list-test-util.h:401] Running for: 30.000s
I20240502 20:45:25.926514 8724 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 20:45:25.926530 8724 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
I20240502 20:45:25.943236 9039 raft_consensus.cc:1261] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Refusing update from remote peer 5da62881d7084cbf848e7b61285e41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:25.943229 9171 raft_consensus.cc:1261] T 273f27ac228d4473ba17b83f75a27ea2 P 5da62881d7084cbf848e7b61285e41a0 [term 1 FOLLOWER]: Refusing update from remote peer 6c3ebe1106e84d3f91c76a45bad4bd24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:25.943868 9248 consensus_queue.cc:1035] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:25.944131 8908 raft_consensus.cc:1261] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Refusing update from remote peer 5da62881d7084cbf848e7b61285e41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:25.944671 9040 raft_consensus.cc:1261] T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Refusing update from remote peer 6c3ebe1106e84d3f91c76a45bad4bd24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:25.944737 9252 consensus_queue.cc:1035] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:25.944671 9039 raft_consensus.cc:1261] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Refusing update from remote peer 5da62881d7084cbf848e7b61285e41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:25.945092 9252 consensus_queue.cc:1035] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:25.945115 9248 consensus_queue.cc:1035] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:25.945694 9253 consensus_queue.cc:1035] T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:25.950742 8909 raft_consensus.cc:1261] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Refusing update from remote peer 5da62881d7084cbf848e7b61285e41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:25.951153 9252 consensus_queue.cc:1035] T 68374b60234f4f30ac844bc877056d6f P 5da62881d7084cbf848e7b61285e41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:25.960427 9269 mvcc.cc:205] Tried to move back new op lower bound from 7023340445462867968 to 7023340444597501952. Current Snapshot: MvccSnapshot[applied={T|T < 7023340445462867968 or (T in {7023340445462867968})}]
I20240502 20:45:25.962085 9268 mvcc.cc:205] Tried to move back new op lower bound from 7023340445462867968 to 7023340444597501952. Current Snapshot: MvccSnapshot[applied={T|T < 7023340445462867968}]
I20240502 20:45:25.963044 9275 mvcc.cc:205] Tried to move back new op lower bound from 7023340445457002496 to 7023340444601868288. Current Snapshot: MvccSnapshot[applied={T|T < 7023340445457002496}]
I20240502 20:45:25.963724 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:28.218633 9262 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:28.305778 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:28.372453 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:29.203712 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:29.327569 9218 maintenance_manager.cc:419] P 5da62881d7084cbf848e7b61285e41a0: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:29.400374 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:29.653435 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.449s user 0.266s sys 0.006s Metrics: {"bytes_written":146898,"cfile_init":1,"compiler_manager_pool.queue_time_us":21736,"dirs.queue_time_us":61373,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":5890,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1993,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8131,"thread_start_us":161,"threads_started":2}
I20240502 20:45:29.654723 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 142699 bytes on disk
I20240502 20:45:29.655436 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 20:45:29.868372 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.467s user 0.280s sys 0.008s Metrics: {"bytes_written":153482,"cfile_init":1,"compiler_manager_pool.queue_time_us":46943,"dirs.queue_time_us":14148,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":822,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1615,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8525,"thread_start_us":185,"threads_started":2}
I20240502 20:45:29.869693 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 146952 bytes on disk
I20240502 20:45:29.870407 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 20:45:30.187860 9108 maintenance_manager.cc:643] P 5da62881d7084cbf848e7b61285e41a0: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.860s user 0.291s sys 0.004s Metrics: {"bytes_written":151452,"cfile_init":1,"compiler_manager_pool.queue_time_us":201284,"compiler_manager_pool.run_cpu_time_us":258492,"compiler_manager_pool.run_wall_time_us":812342,"dirs.queue_time_us":15021,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1067,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1589,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8396,"thread_start_us":169,"threads_started":2}
I20240502 20:45:30.189396 9218 maintenance_manager.cc:419] P 5da62881d7084cbf848e7b61285e41a0: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 145496 bytes on disk
I20240502 20:45:30.203119 9108 maintenance_manager.cc:643] P 5da62881d7084cbf848e7b61285e41a0: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.013s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 20:45:31.024628 9262 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:31.085072 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:31.148350 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:33.127410 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:33.166098 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:33.231734 9218 maintenance_manager.cc:419] P 5da62881d7084cbf848e7b61285e41a0: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:33.372954 9262 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:33.475502 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:33.644865 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:33.704963 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.539s user 0.245s sys 0.004s Metrics: {"bytes_written":15402,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1290,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1018,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1706,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8579,"thread_start_us":78,"threads_started":1,"update_count":2391}
I20240502 20:45:33.706215 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 133085 bytes on disk
I20240502 20:45:33.706676 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 20:45:33.707262 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014814
I20240502 20:45:33.900849 9108 maintenance_manager.cc:643] P 5da62881d7084cbf848e7b61285e41a0: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.669s user 0.253s sys 0.000s Metrics: {"bytes_written":16034,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":21441,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1748,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3169,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8413,"thread_start_us":75,"threads_started":1,"update_count":2490}
I20240502 20:45:33.902050 9218 maintenance_manager.cc:419] P 5da62881d7084cbf848e7b61285e41a0: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 129069 bytes on disk
I20240502 20:45:33.902498 9108 maintenance_manager.cc:643] P 5da62881d7084cbf848e7b61285e41a0: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 20:45:33.903112 9218 maintenance_manager.cc:419] P 5da62881d7084cbf848e7b61285e41a0: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014814
I20240502 20:45:34.125332 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.418s user 0.370s sys 0.016s Metrics: {"bytes_written":300970,"cfile_cache_hit":13,"cfile_cache_hit_bytes":96267,"cfile_cache_miss":63,"cfile_cache_miss_bytes":1008905,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":99,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":830,"drs_written":1,"lbm_read_time_us":2101,"lbm_reads_lt_1ms":95,"lbm_write_time_us":2462,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":547162,"rows_written":16710,"thread_start_us":404,"threads_started":7}
I20240502 20:45:34.126077 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 306433 bytes on disk
I20240502 20:45:34.126845 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240502 20:45:34.549998 8977 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 1.051s user 0.001s sys 0.000s
I20240502 20:45:34.551843 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.424s user 0.209s sys 0.000s Metrics: {"bytes_written":149236,"cfile_init":1,"dirs.queue_time_us":13898,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1515,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":8122,"thread_start_us":83,"threads_started":1}
I20240502 20:45:34.553541 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014817
I20240502 20:45:34.757395 9108 maintenance_manager.cc:643] P 5da62881d7084cbf848e7b61285e41a0: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.854s user 0.398s sys 0.008s Metrics: {"bytes_written":302625,"cfile_cache_hit":14,"cfile_cache_hit_bytes":148643,"cfile_cache_miss":62,"cfile_cache_miss_bytes":958873,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":9033,"drs_written":1,"lbm_read_time_us":2040,"lbm_reads_lt_1ms":94,"lbm_write_time_us":2598,"lbm_writes_lt_1ms":52,"num_input_rowsets":2,"peak_mem_usage":545751,"rows_written":16809,"spinlock_wait_cycles":1408,"thread_start_us":496,"threads_started":8}
I20240502 20:45:34.758479 9218 maintenance_manager.cc:419] P 5da62881d7084cbf848e7b61285e41a0: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 308361 bytes on disk
I20240502 20:45:34.759057 9108 maintenance_manager.cc:643] P 5da62881d7084cbf848e7b61285e41a0: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 20:45:35.116551 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.560s user 0.431s sys 0.004s Metrics: {"bytes_written":300122,"cfile_cache_hit":14,"cfile_cache_hit_bytes":195863,"cfile_cache_miss":54,"cfile_cache_miss_bytes":869285,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":842,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":7565,"drs_written":1,"lbm_read_time_us":1868,"lbm_reads_lt_1ms":86,"lbm_write_time_us":3325,"lbm_writes_lt_1ms":51,"num_input_rowsets":2,"peak_mem_usage":496236,"rows_written":16647,"thread_start_us":655,"threads_started":6}
I20240502 20:45:35.117239 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 305288 bytes on disk
I20240502 20:45:35.117760 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 20:45:35.914785 9279 log.cc:924] Time spent T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20240502 20:45:35.914793 9274 log.cc:924] Time spent T 273f27ac228d4473ba17b83f75a27ea2 P 023da3a68b244e89b00248344f835b3a: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20240502 20:45:35.915179 9276 log.cc:924] Time spent T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20240502 20:45:35.915400 9278 log.cc:924] Time spent T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20240502 20:45:35.915735 9254 log.cc:924] Time spent T 2567a0ea66a441138ca0d35f9c477955 P 5da62881d7084cbf848e7b61285e41a0: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20240502 20:45:35.916076 9258 log.cc:924] Time spent T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20240502 20:45:35.916818 9272 log.cc:924] Time spent T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240502 20:45:35.935123 8724 linked_list-test-util.h:417] Sample at HT timestamp: 7023340486357954560 Inserted count: 32500
I20240502 20:45:36.345165 9262 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:36.421911 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 20:45:36.473383 9262 monotime.cc:342] Check failed: t_beg.Initialized()
*** Check failure stack trace: ***
@ 0x7fa53dd6ddcd google::LogMessage::Fail() at ??:0
@ 0x7fa53dd71b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa53dd6d7cc google::LogMessage::Flush() at ??:0
@ 0x7fa53dd6ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa53f14f5b1 kudu::operator-() at ??:0
@ 0x7fa541232fd2 kudu::tablet::WriteOp::ToString() at ??:0
@ 0x7fa54121c05b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
@ 0x7fa54121bf90 kudu::tablet::OpDriver::ToString() at ??:0
@ 0x7fa5411f29bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
@ 0x7fa541e1e861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
@ 0x7fa541e1d93a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
@ 0x7fa541e26149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
@ 0x7fa53cce8d09 std::function<>::operator()() at ??:0
@ 0x7fa53cce3107 kudu::Webserver::RunPathHandler() at ??:0
@ 0x7fa53cce2a38 kudu::Webserver::BeginRequestCallback() at ??:0
@ 0x7fa53cce2123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
@ 0x7fa53cd0ab89 handle_request at ??:0
@ 0x7fa53cd0d26b worker_thread at ??:0
@ 0x7fa53f9ec6db start_thread at ??:0
@ 0x7fa53cf2471f clone at ??:0
W20240502 20:45:36.684545 9220 meta_cache.cc:302] tablet 2567a0ea66a441138ca0d35f9c477955: replica 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481) has failed: Network error: recv got EOF from 127.8.133.3:42481 (error 108)
W20240502 20:45:36.691475 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.691476 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.697607 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.697607 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
I20240502 20:45:36.699831 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:45:36.710963 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.711002 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.720613 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.721544 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.743077 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.744052 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.753681 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.758064 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.780817 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.786249 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.796950 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.800057 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.829851 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.830785 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.849656 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.850471 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.886758 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.887948 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.907994 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.907951 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.950753 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.953200 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:36.961706 8842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
W20240502 20:45:36.962568 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:45:36.974880 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:36.974960 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.025485 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.026470 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.051478 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.053507 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.112198 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.113438 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.141463 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.142158 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.203842 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.203842 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.237035 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.237867 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.307479 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.308674 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.343544 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.344413 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.418103 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.420361 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.457165 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.459195 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.470777 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.475708 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.477802 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.490779 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.490942 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.499858 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.519183 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.529692 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.534765 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.538926 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.543287 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.553292 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.564558 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.568064 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.578541 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.586429 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.597270 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.611574 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.615324 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.650441 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.664117 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.664117 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.669000 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.671025 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.688690 9220 meta_cache.cc:302] tablet 68374b60234f4f30ac844bc877056d6f: replica 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481) has failed: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
W20240502 20:45:37.708334 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.710319 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.711426 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.716912 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.733824 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.735023 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.784171 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.806110 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.806110 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.809019 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.854934 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.855988 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.864545 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.864622 8868 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
I20240502 20:45:37.875048 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:45:37.891525 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.892527 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
I20240502 20:45:37.913439 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:45:37.955263 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.956120 9000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
W20240502 20:45:37.956301 8999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49658: Illegal state: replica 023da3a68b244e89b00248344f835b3a is not leader of this config: current role FOLLOWER
I20240502 20:45:37.963462 9396 raft_consensus.cc:482] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5da62881d7084cbf848e7b61285e41a0)
I20240502 20:45:37.963625 9396 raft_consensus.cc:504] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:37.964155 9396 leader_election.cc:288] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163), 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
I20240502 20:45:37.964618 9037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" candidate_term: 2 candidate_status { last_received { term: 1 index: 698 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a" is_pre_election: true
W20240502 20:45:37.965444 8842 leader_election.cc:334] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
I20240502 20:45:37.965513 8842 leader_election.cc:302] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c3ebe1106e84d3f91c76a45bad4bd24; no voters: 023da3a68b244e89b00248344f835b3a, 5da62881d7084cbf848e7b61285e41a0
I20240502 20:45:37.965653 9396 raft_consensus.cc:2726] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:45:37.986421 9399 raft_consensus.cc:482] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5da62881d7084cbf848e7b61285e41a0)
I20240502 20:45:37.986538 9399 raft_consensus.cc:504] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:37.987895 9399 leader_election.cc:288] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481), 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619)
W20240502 20:45:37.988435 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:37.988890 8973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
W20240502 20:45:37.989780 8973 leader_election.cc:334] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
I20240502 20:45:37.992408 8909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "023da3a68b244e89b00248344f835b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 698 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" is_pre_election: true
I20240502 20:45:37.992532 8909 raft_consensus.cc:2454] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023da3a68b244e89b00248344f835b3a in term 1.
I20240502 20:45:37.992871 8975 leader_election.cc:302] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters: 5da62881d7084cbf848e7b61285e41a0
I20240502 20:45:37.993085 9399 raft_consensus.cc:2781] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:37.993160 9399 raft_consensus.cc:482] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5da62881d7084cbf848e7b61285e41a0)
I20240502 20:45:37.993201 9399 raft_consensus.cc:3037] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:37.996332 9396 raft_consensus.cc:482] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5da62881d7084cbf848e7b61285e41a0)
I20240502 20:45:37.996440 9396 raft_consensus.cc:504] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:37.996863 9396 leader_election.cc:288] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 023da3a68b244e89b00248344f835b3a (127.8.133.2:36163), 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
I20240502 20:45:37.997249 9037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" candidate_term: 2 candidate_status { last_received { term: 1 index: 686 } } ignore_live_leader: false dest_uuid: "023da3a68b244e89b00248344f835b3a" is_pre_election: true
W20240502 20:45:37.997874 8842 leader_election.cc:334] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
I20240502 20:45:37.997946 8842 leader_election.cc:302] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c3ebe1106e84d3f91c76a45bad4bd24; no voters: 023da3a68b244e89b00248344f835b3a, 5da62881d7084cbf848e7b61285e41a0
I20240502 20:45:37.998098 9396 raft_consensus.cc:2726] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:45:38.002633 9404 raft_consensus.cc:482] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5da62881d7084cbf848e7b61285e41a0)
I20240502 20:45:38.002739 9404 raft_consensus.cc:504] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:38.003140 9404 leader_election.cc:288] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619), 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
I20240502 20:45:38.003566 8909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "023da3a68b244e89b00248344f835b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 686 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" is_pre_election: true
I20240502 20:45:38.003707 8909 raft_consensus.cc:2454] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023da3a68b244e89b00248344f835b3a in term 1.
I20240502 20:45:38.004032 8975 leader_election.cc:302] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters:
W20240502 20:45:38.004209 8973 leader_election.cc:334] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
I20240502 20:45:38.004215 9404 raft_consensus.cc:2781] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:38.004299 9404 raft_consensus.cc:482] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5da62881d7084cbf848e7b61285e41a0)
I20240502 20:45:38.004380 9404 raft_consensus.cc:3037] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:45:38.006529 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:38.007562 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
W20240502 20:45:38.019145 8869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45314: Illegal state: replica 6c3ebe1106e84d3f91c76a45bad4bd24 is not leader of this config: current role FOLLOWER
I20240502 20:45:38.019136 9399 raft_consensus.cc:504] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:38.019621 9399 leader_election.cc:288] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 election: Requested vote from peers 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481), 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619)
I20240502 20:45:38.020423 8909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2567a0ea66a441138ca0d35f9c477955" candidate_uuid: "023da3a68b244e89b00248344f835b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 698 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
I20240502 20:45:38.020546 8909 raft_consensus.cc:3037] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:38.020907 9404 raft_consensus.cc:504] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:38.021713 8909 raft_consensus.cc:2454] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023da3a68b244e89b00248344f835b3a in term 2.
I20240502 20:45:38.021848 9404 leader_election.cc:288] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 election: Requested vote from peers 6c3ebe1106e84d3f91c76a45bad4bd24 (127.8.133.1:46619), 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481)
W20240502 20:45:38.022130 8973 leader_election.cc:334] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
I20240502 20:45:38.022236 8975 leader_election.cc:302] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters: 5da62881d7084cbf848e7b61285e41a0
I20240502 20:45:38.022529 8909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68374b60234f4f30ac844bc877056d6f" candidate_uuid: "023da3a68b244e89b00248344f835b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 686 } } ignore_live_leader: false dest_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24"
I20240502 20:45:38.022639 8909 raft_consensus.cc:3037] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:45:38.022745 8973 leader_election.cc:334] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111)
I20240502 20:45:38.022902 9404 raft_consensus.cc:2781] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:38.022979 9404 raft_consensus.cc:686] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [term 2 LEADER]: Becoming Leader. State: Replica: 023da3a68b244e89b00248344f835b3a, State: Running, Role: LEADER
I20240502 20:45:38.023322 9404 consensus_queue.cc:237] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 698, Committed index: 698, Last appended: 1.698, Last appended by leader: 698, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } }
I20240502 20:45:38.023759 8909 raft_consensus.cc:2454] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023da3a68b244e89b00248344f835b3a in term 2.
I20240502 20:45:38.024341 8975 leader_election.cc:302] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 023da3a68b244e89b00248344f835b3a, 6c3ebe1106e84d3f91c76a45bad4bd24; no voters: 5da62881d7084cbf848e7b61285e41a0
I20240502 20:45:38.024495 9404 raft_consensus.cc:2781] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:38.024554 9404 raft_consensus.cc:686] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [term 2 LEADER]: Becoming Leader. State: Replica: 023da3a68b244e89b00248344f835b3a, State: Running, Role: LEADER
I20240502 20:45:38.024677 9404 consensus_queue.cc:237] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 686, Committed index: 686, Last appended: 1.686, Last appended by leader: 686, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } }
I20240502 20:45:38.025987 8766 catalog_manager.cc:5484] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a reported cstate change: term changed from 1 to 2, leader changed from 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3) to 023da3a68b244e89b00248344f835b3a (127.8.133.2). New cstate: current_term: 2 leader_uuid: "023da3a68b244e89b00248344f835b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:38.029958 8766 catalog_manager.cc:5484] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a reported cstate change: term changed from 1 to 2, leader changed from 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3) to 023da3a68b244e89b00248344f835b3a (127.8.133.2). New cstate: current_term: 2 leader_uuid: "023da3a68b244e89b00248344f835b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023da3a68b244e89b00248344f835b3a" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5da62881d7084cbf848e7b61285e41a0" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 42481 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:38.077735 8909 raft_consensus.cc:1261] T 2567a0ea66a441138ca0d35f9c477955 P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 2 FOLLOWER]: Refusing update from remote peer 023da3a68b244e89b00248344f835b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 698. Preceding OpId from leader: term: 2 index: 700. (index mismatch)
I20240502 20:45:38.078388 9399 consensus_queue.cc:1035] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 699, Last known committed idx: 698, Time since last communication: 0.000s
W20240502 20:45:38.078732 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:38.094553 8909 raft_consensus.cc:1261] T 68374b60234f4f30ac844bc877056d6f P 6c3ebe1106e84d3f91c76a45bad4bd24 [term 2 FOLLOWER]: Refusing update from remote peer 023da3a68b244e89b00248344f835b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 686. Preceding OpId from leader: term: 2 index: 688. (index mismatch)
I20240502 20:45:38.095017 9399 consensus_queue.cc:1035] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3ebe1106e84d3f91c76a45bad4bd24" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 687, Last known committed idx: 686, Time since last communication: 0.000s
W20240502 20:45:38.095927 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:38.627477 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:38.637444 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:38.899122 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.271s user 0.198s sys 0.004s Metrics: {"bytes_written":147724,"cfile_init":1,"dirs.queue_time_us":11535,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1026,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1561,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8004,"thread_start_us":74,"threads_started":1}
I20240502 20:45:38.899860 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014704
I20240502 20:45:39.050223 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.413s user 0.202s sys 0.000s Metrics: {"bytes_written":147147,"cfile_init":1,"dirs.queue_time_us":27646,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1279,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1366,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7973,"thread_start_us":80,"threads_started":1}
I20240502 20:45:39.051126 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 127665 bytes on disk
I20240502 20:45:39.051890 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 20:45:39.052539 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014704
W20240502 20:45:39.260452 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:45:39.491469 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:39.551607 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:40.034502 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.134s user 0.549s sys 0.004s Metrics: {"bytes_written":10889,"cfile_cache_hit":11,"cfile_cache_hit_bytes":47888,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1590276,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":767,"drs_written":1,"lbm_read_time_us":2698,"lbm_reads_lt_1ms":107,"lbm_write_time_us":3302,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":785454,"reinsert_count":0,"rows_written":24651,"thread_start_us":354,"threads_started":7,"update_count":1683}
I20240502 20:45:40.035344 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 446386 bytes on disk
I20240502 20:45:40.035903 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 20:45:40.036442 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.028087
I20240502 20:45:40.095753 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.059s user 0.057s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":14104,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":66,"lbm_read_time_us":225,"lbm_reads_lt_1ms":20,"lbm_write_time_us":226,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1683}
I20240502 20:45:40.213193 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.160s user 0.553s sys 0.008s Metrics: {"bytes_written":11074,"cfile_cache_hit":11,"cfile_cache_hit_bytes":47671,"cfile_cache_miss":75,"cfile_cache_miss_bytes":1591363,"cfile_init":7,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":9748,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1071,"drs_written":1,"lbm_read_time_us":6041,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":102,"lbm_write_time_us":3400,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":785965,"reinsert_count":0,"rows_written":24683,"thread_start_us":405,"threads_started":7,"update_count":1715}
I20240502 20:45:40.218492 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 446757 bytes on disk
I20240502 20:45:40.221082 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 20:45:40.221709 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.028532
W20240502 20:45:40.271894 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:45:40.300186 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.078s user 0.057s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":14317,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":93,"lbm_read_time_us":265,"lbm_reads_lt_1ms":20,"lbm_write_time_us":259,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1715}
W20240502 20:45:40.597606 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:45:41.303892 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2): perf score=1.000000
I20240502 20:45:41.348508 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2): perf score=1.000000
I20240502 20:45:41.354609 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:41.440762 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:41.586194 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.282s user 0.209s sys 0.000s Metrics: {"bytes_written":142233,"cfile_init":1,"dirs.queue_time_us":15787,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":846,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1685,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7769,"thread_start_us":124,"threads_started":1}
I20240502 20:45:41.587142 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(68374b60234f4f30ac844bc877056d6f): perf score=1.000000
I20240502 20:45:41.700155 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.351s user 0.216s sys 0.011s Metrics: {"bytes_written":142911,"cfile_init":1,"dirs.queue_time_us":12984,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1050,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1948,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7802,"thread_start_us":78,"threads_started":1}
I20240502 20:45:41.701021 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(68374b60234f4f30ac844bc877056d6f): perf score=1.000000
W20240502 20:45:41.721031 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:45:41.814090 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.227s user 0.206s sys 0.000s Metrics: {"bytes_written":143375,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":853,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1372,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7819}
I20240502 20:45:41.814790 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f): 146391 bytes on disk
I20240502 20:45:41.815291 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:45:41.815908 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2): 146315 bytes on disk
I20240502 20:45:41.816283 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 20:45:41.926983 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.224s user 0.212s sys 0.000s Metrics: {"bytes_written":144272,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":880,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1424,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7877}
I20240502 20:45:41.927681 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f): 146982 bytes on disk
I20240502 20:45:41.928110 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 20:45:41.928797 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2): 146710 bytes on disk
I20240502 20:45:41.929216 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 20:45:42.170647 8842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 20:45:42.180087 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:42.317809 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:42.536665 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.356s user 0.230s sys 0.000s Metrics: {"bytes_written":14228,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":22528,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1187,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1652,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8218,"thread_start_us":75,"threads_started":1,"update_count":2203}
I20240502 20:45:42.537520 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014571
I20240502 20:45:42.764366 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.446s user 0.234s sys 0.007s Metrics: {"bytes_written":14224,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13684,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":2824,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1547,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8317,"thread_start_us":88,"threads_started":1,"update_count":2203}
I20240502 20:45:42.767732 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 129087 bytes on disk
I20240502 20:45:42.769171 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 20:45:42.769867 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014570
W20240502 20:45:42.856355 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:42.868654 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:45:43.259994 8973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20240502 20:45:43.523098 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.985s user 0.736s sys 0.020s Metrics: {"bytes_written":593226,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1045,"cfile_cache_miss":100,"cfile_cache_miss_bytes":2221975,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":19872,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":5554,"drs_written":1,"lbm_read_time_us":3681,"lbm_reads_lt_1ms":140,"lbm_write_time_us":4558,"lbm_writes_lt_1ms":81,"num_input_rowsets":2,"peak_mem_usage":1090855,"rows_written":32869,"thread_start_us":468,"threads_started":8}
I20240502 20:45:43.526173 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 623875 bytes on disk
I20240502 20:45:43.526594 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 20:45:43.563262 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:43.619040 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:43.881778 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.111s user 0.813s sys 0.008s Metrics: {"bytes_written":1054,"cfile_cache_hit":13,"cfile_cache_hit_bytes":49920,"cfile_cache_miss":99,"cfile_cache_miss_bytes":2180924,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":13048,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":61649,"drs_written":1,"lbm_read_time_us":3416,"lbm_reads_lt_1ms":135,"lbm_write_time_us":5153,"lbm_writes_lt_1ms":85,"num_input_rowsets":2,"peak_mem_usage":1092875,"reinsert_count":0,"rows_written":33000,"thread_start_us":132,"threads_started":1,"update_count":131}
I20240502 20:45:43.882514 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 625289 bytes on disk
I20240502 20:45:43.882963 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 20:45:43.883630 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.002054
I20240502 20:45:43.973802 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.090s user 0.049s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1619,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":242,"lbm_reads_lt_1ms":20,"lbm_write_time_us":210,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":131}
W20240502 20:45:44.214548 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:45:45.184545 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:45:45.237706 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:45.299803 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:45:45.424499 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:45:45.534754 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:45.729750 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:45.793244 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.258s user 0.211s sys 0.000s Metrics: {"bytes_written":146767,"cfile_init":1,"dirs.queue_time_us":8860,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1625,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8098,"thread_start_us":75,"threads_started":1}
I20240502 20:45:45.794086 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014449
I20240502 20:45:45.939314 8724 linked_list-test-util.h:417] Sample at HT timestamp: 7023340527299510272 Inserted count: 63000
I20240502 20:45:46.139140 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.409s user 0.272s sys 0.004s Metrics: {"bytes_written":152780,"cfile_init":1,"dirs.queue_time_us":13454,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":842,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1650,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8466,"thread_start_us":84,"threads_started":1}
I20240502 20:45:46.140234 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 143658 bytes on disk
I20240502 20:45:46.143817 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 20:45:46.145519 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014446
W20240502 20:45:46.630147 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:45:46.940682 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.146s user 0.891s sys 0.004s Metrics: {"bytes_written":2654,"cfile_cache_hit":11,"cfile_cache_hit_bytes":48796,"cfile_cache_miss":115,"cfile_cache_miss_bytes":2753110,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":15359,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1098,"drs_written":1,"lbm_read_time_us":4105,"lbm_reads_lt_1ms":147,"lbm_write_time_us":5313,"lbm_writes_lt_1ms":97,"num_input_rowsets":2,"peak_mem_usage":1355401,"reinsert_count":0,"rows_written":40967,"thread_start_us":476,"threads_started":7,"update_count":379}
I20240502 20:45:46.941525 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 781519 bytes on disk
I20240502 20:45:46.942013 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 20:45:46.942593 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.004172
I20240502 20:45:47.036136 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.093s user 0.063s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4124,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":2483,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":604,"lbm_read_time_us":246,"lbm_reads_lt_1ms":20,"lbm_write_time_us":210,"lbm_writes_lt_1ms":8,"mutex_wait_us":2403,"reinsert_count":0,"update_count":379}
I20240502 20:45:47.334061 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:47.398881 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:47.450879 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.305s user 0.991s sys 0.007s Metrics: {"bytes_written":5793,"cfile_cache_hit":14,"cfile_cache_hit_bytes":419084,"cfile_cache_miss":118,"cfile_cache_miss_bytes":2408472,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":10422,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":881,"drs_written":1,"lbm_read_time_us":3897,"lbm_reads_lt_1ms":150,"lbm_write_time_us":5437,"lbm_writes_lt_1ms":98,"num_input_rowsets":2,"peak_mem_usage":1363122,"reinsert_count":0,"rows_written":41466,"thread_start_us":503,"threads_started":7,"update_count":878}
I20240502 20:45:47.452212 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.008979
I20240502 20:45:47.537132 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.084s user 0.065s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":8648,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":100,"lbm_read_time_us":261,"lbm_reads_lt_1ms":24,"lbm_write_time_us":203,"lbm_writes_lt_1ms":8,"reinsert_count":0,"spinlock_wait_cycles":768,"update_count":878}
I20240502 20:45:47.537838 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 787171 bytes on disk
I20240502 20:45:47.538228 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240502 20:45:47.599625 8842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 20:45:47.660988 9461 consensus_queue.cc:786] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Peer 5da62881d7084cbf848e7b61285e41a0 is lagging by at least 11 ops behind the committed index
W20240502 20:45:47.670121 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:47.886771 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:45:48.280874 8973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 20:45:48.983990 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:49.036513 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:49.040112 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
W20240502 20:45:49.070977 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:45:49.292225 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.252s user 0.209s sys 0.000s Metrics: {"bytes_written":152194,"cfile_init":1,"dirs.queue_time_us":4273,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":996,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1453,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8296,"thread_start_us":92,"threads_started":1}
I20240502 20:45:49.293485 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014449
I20240502 20:45:49.294620 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:49.514032 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.219s user 0.160s sys 0.004s Metrics: {"bytes_written":154296,"cfile_init":1,"dirs.queue_time_us":214,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1207,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":8430,"thread_start_us":93,"threads_started":1}
I20240502 20:45:49.514997 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 133334 bytes on disk
I20240502 20:45:49.515525 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 20:45:49.516253 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014372
W20240502 20:45:50.253613 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:45:50.404289 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:45:50.651669 9535 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49658 (ReqId={client: 669b81d6ba714c3c83cf6ef0d4514c65, seq_no=4526, attempt_no=0}) took 1052 ms. Trace:
I20240502 20:45:50.653326 9535 rpcz_store.cc:276] 0502 20:45:49.599130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 20:45:49.599191 (+ 61us) service_pool.cc:225] Handling call
0502 20:45:50.651149 (+1051958us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 20:45:49.599597 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2567a0ea66a441138ca0d35f9c477955
0502 20:45:49.599686 (+ 89us) write_op.cc:424] Acquiring schema lock in shared mode
0502 20:45:49.599689 (+ 3us) write_op.cc:427] Acquired schema lock
0502 20:45:49.599691 (+ 2us) tablet.cc:660] Decoding operations
0502 20:45:49.608179 (+ 8488us) write_op.cc:612] Acquiring the partition lock for write op
0502 20:45:49.608188 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0502 20:45:49.608191 (+ 3us) tablet.cc:683] Acquiring locks for 2181 operations
0502 20:45:49.628985 (+ 20794us) tablet.cc:699] Row locks acquired
0502 20:45:49.628988 (+ 3us) write_op.cc:260] PREPARE: finished
0502 20:45:49.629029 (+ 41us) write_op.cc:270] Start()
0502 20:45:49.629060 (+ 31us) write_op.cc:276] Timestamp: P: 1714682749629010 usec, L: 0
0502 20:45:49.629062 (+ 2us) op_driver.cc:331] REPLICATION: starting
0502 20:45:49.629200 (+ 138us) log.cc:842] Serialized 24341 byte log entry
0502 20:45:49.637112 (+ 7912us) op_driver.cc:444] REPLICATION: finished
0502 20:45:49.648804 (+ 11692us) write_op.cc:301] APPLY: starting
0502 20:45:49.648814 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0502 20:45:50.042769 (+393955us) tablet.cc:1368] finished BulkCheckPresence
0502 20:45:50.042783 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 20:45:50.642411 (+599628us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 20:45:50.643346 (+ 935us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=5785,key_file_lookups=2846,delta_file_lookups=0,mrs_lookups=758
0502 20:45:50.643352 (+ 6us) write_op.cc:312] APPLY: finished
0502 20:45:50.645969 (+ 2617us) log.cc:842] Serialized 15952 byte log entry
0502 20:45:50.646150 (+ 181us) write_op.cc:481] Releasing partition, row and schema locks
0502 20:45:50.650981 (+ 4831us) write_op.cc:446] Released schema lock
0502 20:45:50.651090 (+ 109us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11652,"cfile_cache_hit":6883,"cfile_cache_hit_bytes":143482904,"cfile_cache_miss":2,"cfile_cache_miss_bytes":48713,"lbm_read_time_us":62,"lbm_reads_lt_1ms":2,"mutex_wait_us":11505,"num_ops":2181,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":17923,"prepare.run_wall_time_us":30156,"raft.queue_time_us":55,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"replication_time_us":8034,"spinlock_wait_cycles":3968}]]}
I20240502 20:45:50.853665 9461 consensus_queue.cc:786] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Peer 5da62881d7084cbf848e7b61285e41a0 is lagging by at least 244 ops behind the committed index [suppressed 5 similar messages]
I20240502 20:45:51.186990 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.893s user 1.069s sys 0.019s Metrics: {"bytes_written":9225,"cfile_cache_hit":12,"cfile_cache_hit_bytes":50033,"cfile_cache_miss":134,"cfile_cache_miss_bytes":3321079,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1225,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":7484,"drs_written":1,"lbm_read_time_us":5842,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":169,"lbm_write_time_us":8056,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":109,"num_input_rowsets":2,"peak_mem_usage":1614525,"reinsert_count":0,"rows_written":49263,"thread_start_us":471,"threads_started":7,"update_count":1423}
I20240502 20:45:51.189097 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.011675
I20240502 20:45:51.315950 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.127s user 0.091s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":13447,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":50,"lbm_read_time_us":337,"lbm_reads_lt_1ms":24,"lbm_write_time_us":451,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":1423}
I20240502 20:45:51.316709 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 937105 bytes on disk
I20240502 20:45:51.319701 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 20:45:51.633442 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 2.117s user 1.168s sys 0.020s Metrics: {"bytes_written":13264,"cfile_cache_hit":16,"cfile_cache_hit_bytes":513582,"cfile_cache_miss":130,"cfile_cache_miss_bytes":2888950,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1778,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1596,"drs_written":1,"lbm_read_time_us":11587,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":161,"lbm_write_time_us":6808,"lbm_writes_lt_1ms":112,"num_input_rowsets":2,"peak_mem_usage":1624289,"reinsert_count":0,"rows_written":49896,"thread_start_us":501,"threads_started":7,"update_count":2056}
I20240502 20:45:51.634359 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 944487 bytes on disk
I20240502 20:45:51.644944 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:45:51.645614 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.016554
I20240502 20:45:51.758023 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.112s user 0.090s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":18503,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":67,"lbm_read_time_us":268,"lbm_reads_lt_1ms":21,"lbm_write_time_us":263,"lbm_writes_lt_1ms":9,"reinsert_count":0,"update_count":2056}
I20240502 20:45:51.877218 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:45:51.880268 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:51.938325 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:45:52.649291 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:52.823251 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
W20240502 20:45:52.984339 8842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:45:52.990140 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:45:53.012717 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.000000
I20240502 20:45:53.181591 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.358s user 0.248s sys 0.005s Metrics: {"bytes_written":20184,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13656,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":2445,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2770,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8814,"thread_start_us":91,"threads_started":1,"update_count":3151}
I20240502 20:45:53.182511 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014402
I20240502 20:45:53.339445 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.327s user 0.225s sys 0.000s Metrics: {"bytes_written":153978,"cfile_init":1,"dirs.queue_time_us":16685,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":750,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1515,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":8501,"thread_start_us":100,"threads_started":1}
I20240502 20:45:53.340404 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 148064 bytes on disk
I20240502 20:45:53.343637 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 20:45:53.344334 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955): perf score=1.014398
W20240502 20:45:53.506696 8973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:45:53.628021 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:53.680497 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:53.984464 9539 consensus_queue.cc:786] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Peer 5da62881d7084cbf848e7b61285e41a0 is lagging by at least 463 ops behind the committed index [suppressed 5 similar messages]
W20240502 20:45:54.681408 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:45:54.750211 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.563s user 1.313s sys 0.009s Metrics: {"bytes_written":948,"cfile_cache_hit":15,"cfile_cache_hit_bytes":417251,"cfile_cache_miss":153,"cfile_cache_miss_bytes":3623103,"cfile_init":10,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":13130,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":30198,"drs_written":1,"lbm_read_time_us":5427,"lbm_reads_lt_1ms":193,"lbm_write_time_us":8755,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":124,"num_input_rowsets":2,"peak_mem_usage":1933122,"reinsert_count":0,"rows_written":58077,"spinlock_wait_cycles":768,"thread_start_us":505,"threads_started":8,"update_count":116}
I20240502 20:45:54.751024 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2): perf score=1.000000
I20240502 20:45:54.887199 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 1.543s user 1.294s sys 0.012s Metrics: {"bytes_written":3024,"cfile_cache_hit":16,"cfile_cache_hit_bytes":605423,"cfile_cache_miss":150,"cfile_cache_miss_bytes":3462887,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":18363,"dirs.run_cpu_time_us":522,"dirs.run_wall_time_us":3120,"drs_written":1,"lbm_read_time_us":6960,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":181,"lbm_write_time_us":7433,"lbm_writes_lt_1ms":126,"mutex_wait_us":1867,"num_input_rowsets":2,"peak_mem_usage":1938174,"reinsert_count":0,"rows_written":58397,"thread_start_us":416,"threads_started":6,"update_count":436}
I20240502 20:45:54.888075 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2): perf score=1.000000
W20240502 20:45:54.998739 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:45:55.068670 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.317s user 0.205s sys 0.004s Metrics: {"bytes_written":2117,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":816,"drs_written":1,"lbm_read_time_us":1124,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1592,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7666,"update_count":302}
I20240502 20:45:55.069645 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(273f27ac228d4473ba17b83f75a27ea2): perf score=1.014711
W20240502 20:45:55.284731 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:45:55.298118 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.406s user 0.202s sys 0.008s Metrics: {"bytes_written":144894,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":763,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1481,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7745}
I20240502 20:45:55.298838 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 1127661 bytes on disk
I20240502 20:45:55.299178 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 20:45:55.299655 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(273f27ac228d4473ba17b83f75a27ea2): perf score=1.014710
I20240502 20:45:55.467379 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.397s user 0.338s sys 0.000s Metrics: {"bytes_written":288055,"cfile_cache_hit":13,"cfile_cache_hit_bytes":88818,"cfile_cache_miss":59,"cfile_cache_miss_bytes":984914,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":804,"drs_written":1,"lbm_read_time_us":1606,"lbm_reads_lt_1ms":95,"lbm_write_time_us":2297,"lbm_writes_lt_1ms":50,"num_input_rowsets":2,"peak_mem_usage":506321,"rows_written":15468,"thread_start_us":479,"threads_started":7}
I20240502 20:45:55.470088 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling FlushMRSOp(68374b60234f4f30ac844bc877056d6f): perf score=1.000000
I20240502 20:45:55.695523 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: FlushMRSOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.225s user 0.206s sys 0.004s Metrics: {"bytes_written":146061,"cfile_init":1,"dirs.queue_time_us":535,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1727,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1367,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":7768}
I20240502 20:45:55.696763 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955): 1123950 bytes on disk
I20240502 20:45:55.697193 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 20:45:55.697763 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2): 300309 bytes on disk
I20240502 20:45:55.707691 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 20:45:55.708261 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling CompactRowSetsOp(68374b60234f4f30ac844bc877056d6f): perf score=1.014761
I20240502 20:45:55.794433 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.495s user 0.348s sys 0.004s Metrics: {"bytes_written":288839,"cfile_cache_hit":12,"cfile_cache_hit_bytes":88299,"cfile_cache_miss":56,"cfile_cache_miss_bytes":989257,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":11483,"dirs.run_cpu_time_us":563,"dirs.run_wall_time_us":9720,"drs_written":1,"lbm_read_time_us":1769,"lbm_reads_lt_1ms":88,"lbm_write_time_us":2147,"lbm_writes_lt_1ms":50,"mutex_wait_us":4980,"num_input_rowsets":2,"peak_mem_usage":508311,"rows_written":15514,"thread_start_us":8102,"threads_started":7}
I20240502 20:45:55.795934 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2): 301095 bytes on disk
I20240502 20:45:55.800426 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(273f27ac228d4473ba17b83f75a27ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 20:45:55.801149 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling FlushMRSOp(68374b60234f4f30ac844bc877056d6f): perf score=1.000000
I20240502 20:45:55.802625 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:55.861038 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:55.945211 8724 linked_list-test-util.h:417] Sample at HT timestamp: 7023340568167927808 Inserted count: 94850
I20240502 20:45:55.945375 8724 linked_list-test-util.h:421] Finished inserting list. Added 94850 in chain
I20240502 20:45:55.945886 8724 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 20:45:55.945991 8724 linked_list-test-util.h:424] 0: -6131359613211967488
I20240502 20:45:55.946062 8724 linked_list-test-util.h:424] 1: -4124606154859282431
I20240502 20:45:55.946179 8724 linked_list-test-util.h:424] 2: -2117711959018176510
I20240502 20:45:55.946252 8724 linked_list-test-util.h:424] 3: -111099238153781245
I20240502 20:45:55.946314 8724 linked_list-test-util.h:424] 4: -4235564655524642812
I20240502 20:45:55.946377 8724 linked_list-test-util.h:424] 5: -2228951934660247547
I20240502 20:45:55.946437 8724 linked_list-test-util.h:424] 6: -222057738819141626
I20240502 20:45:55.946501 8724 linked_list-test-util.h:424] 7: 1784554982045253639
I20240502 20:45:55.946563 8724 linked_list-test-util.h:424] 8: -2339769697837318136
I20240502 20:45:55.946624 8724 linked_list-test-util.h:424] 9: -333156976972922871
I20240502 20:45:55.946689 8724 linked_list-test-util.h:424] 10: 1673737218868183050
I20240502 20:45:55.946749 8724 linked_list-test-util.h:424] 11: 3680349939732578315
I20240502 20:45:55.946971 8724 linked_list-test-util.h:424] 12: -444115477638283252
I20240502 20:45:55.947048 8724 linked_list-test-util.h:424] 13: 1562778718202822669
I20240502 20:45:55.947111 8724 linked_list-test-util.h:424] 14: 3569391439067217934
I20240502 20:45:55.947269 8724 linked_list-test-util.h:424] 15: 5576144897419902991
I20240502 20:45:55.947350 8724 linked_list-test-util.h:424] 16: 1451679480049041424
I20240502 20:45:55.951303 8724 linked_list-test-util.h:424] 17: 3458573675890147345
I20240502 20:45:55.951408 8724 linked_list-test-util.h:424] 18: 5465186396754542610
I20240502 20:45:55.951471 8724 linked_list-test-util.h:424] 19: 7471799117618937875
I20240502 20:45:55.951527 8724 linked_list-test-util.h:424] 20: 3347333700248076308
I20240502 20:45:55.951587 8724 linked_list-test-util.h:424] 21: 5354227896089182229
I20240502 20:45:55.951687 8724 linked_list-test-util.h:424] 22: 7360981354441867286
I20240502 20:45:55.951746 8724 linked_list-test-util.h:424] 23: -9079149998403289065
I20240502 20:45:55.951818 8724 linked_list-test-util.h:424] 24: 5243128657935400984
I20240502 20:45:55.951875 8724 linked_list-test-util.h:424] 25: 7250022853776506905
I20240502 20:45:55.951932 8724 linked_list-test-util.h:424] 26: -9190108499068649446
I20240502 20:45:55.951988 8724 linked_list-test-util.h:424] 27: 5132170157270040603
I20240502 20:45:55.952045 8724 linked_list-test-util.h:424] 28: 7138782878134435868
I20240502 20:45:55.952102 8724 linked_list-test-util.h:424] 29: 9145817811463831581
I20240502 20:45:55.952155 8724 linked_list-test-util.h:424] 30: -7294313541381324770
I20240502 20:45:55.952209 8724 linked_list-test-util.h:424] 31: 7027965114957365279
I20240502 20:45:55.952266 8724 linked_list-test-util.h:424] 32: 9034577835821760544
I20240502 20:45:55.952323 8724 linked_list-test-util.h:424] 33: -7405272042046685151
I20240502 20:45:55.952378 8724 linked_list-test-util.h:424] 34: -5398659321182289886
I20240502 20:45:55.952601 8724 linked_list-test-util.h:424] 35: 8923619335156400163
I20240502 20:45:55.952662 8724 linked_list-test-util.h:424] 36: -7516371280200466396
I20240502 20:45:55.952719 8724 linked_list-test-util.h:424] 37: -5509477084359360475
I20240502 20:45:55.952776 8724 linked_list-test-util.h:424] 38: -3502864363494965210
I20240502 20:45:55.952838 8724 linked_list-test-util.h:424] 39: -7627329780865826777
I20240502 20:45:55.952891 8724 linked_list-test-util.h:424] 40: -5620435585024720856
I20240502 20:45:55.952948 8724 linked_list-test-util.h:424] 41: -3613822864160325591
I20240502 20:45:55.953003 8724 linked_list-test-util.h:424] 42: -1607210143295930326
I20240502 20:45:55.953058 8724 linked_list-test-util.h:424] 43: -5731534823178502101
I20240502 20:45:55.953114 8724 linked_list-test-util.h:424] 44: -3724640627337396180
I20240502 20:45:55.953166 8724 linked_list-test-util.h:424] 45: -1718027906473000915
I20240502 20:45:55.953222 8724 linked_list-test-util.h:424] 46: 288584814391394350
I20240502 20:45:55.953277 8724 linked_list-test-util.h:424] 47: -3835880602979467217
I20240502 20:45:55.953332 8724 linked_list-test-util.h:424] 48: -1828986407138361296
I20240502 20:45:55.953388 8724 linked_list-test-util.h:424] 49: 177626313726033969
I20240502 20:45:56.057565 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 1714682735927235 usec, L: 1 Expected Rows: 32500
I20240502 20:45:56.077414 8724 linked_list-test-util.h:528] Calling callback on tserver 023da3a68b244e89b00248344f835b3a
I20240502 20:45:56.119530 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: FlushMRSOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.318s user 0.217s sys 0.000s Metrics: {"bytes_written":2971,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":942,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7261,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"reinsert_count":0,"rows_written":7971,"update_count":434}
I20240502 20:45:56.121145 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f): 148263 bytes on disk
I20240502 20:45:56.121766 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 20:45:56.122907 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling CompactRowSetsOp(68374b60234f4f30ac844bc877056d6f): perf score=1.014759
I20240502 20:45:56.155449 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: CompactRowSetsOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.447s user 0.364s sys 0.008s Metrics: {"bytes_written":2043,"cfile_cache_hit":13,"cfile_cache_hit_bytes":133266,"cfile_cache_miss":55,"cfile_cache_miss_bytes":948266,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":850,"drs_written":1,"lbm_read_time_us":1824,"lbm_reads_lt_1ms":87,"lbm_write_time_us":2458,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":504330,"reinsert_count":0,"rows_written":15645,"spinlock_wait_cycles":768,"thread_start_us":393,"threads_started":7,"update_count":289}
I20240502 20:45:56.156394 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling MajorDeltaCompactionOp(68374b60234f4f30ac844bc877056d6f): perf score=0.008157
I20240502 20:45:56.226224 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: MajorDeltaCompactionOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.070s user 0.028s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":2985,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":270,"lbm_reads_lt_1ms":24,"lbm_write_time_us":212,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":289}
I20240502 20:45:56.228540 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.001024
I20240502 20:45:56.398854 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.170s user 0.078s sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":587,"cfile_cache_miss":2,"cfile_cache_miss_bytes":847,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":10,"lbm_write_time_us":226,"lbm_writes_lt_1ms":8,"reinsert_count":0,"update_count":116}
I20240502 20:45:56.399648 9087 maintenance_manager.cc:419] P 023da3a68b244e89b00248344f835b3a: Scheduling UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f): 301279 bytes on disk
I20240502 20:45:56.400352 8977 maintenance_manager.cc:643] P 023da3a68b244e89b00248344f835b3a: UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:45:56.530282 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: CompactRowSetsOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.404s user 0.310s sys 0.010s Metrics: {"bytes_written":293020,"cfile_cache_hit":13,"cfile_cache_hit_bytes":89012,"cfile_cache_miss":59,"cfile_cache_miss_bytes":1006576,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":1868,"dirs.run_cpu_time_us":294,"dirs.run_wall_time_us":1488,"drs_written":1,"lbm_read_time_us":1687,"lbm_reads_lt_1ms":91,"lbm_write_time_us":2468,"lbm_writes_lt_1ms":50,"mutex_wait_us":223,"num_input_rowsets":2,"peak_mem_usage":515421,"rows_written":15790,"thread_start_us":395,"threads_started":7}
I20240502 20:45:56.531260 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955): perf score=0.003243
I20240502 20:45:56.610388 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: MajorDeltaCompactionOp(2567a0ea66a441138ca0d35f9c477955) complete. Timing: real 0.079s user 0.076s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":4829,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":98,"lbm_read_time_us":230,"lbm_reads_lt_1ms":20,"lbm_write_time_us":238,"lbm_writes_lt_1ms":8,"mutex_wait_us":25,"reinsert_count":0,"spinlock_wait_cycles":1920,"update_count":436}
I20240502 20:45:56.611100 8956 maintenance_manager.cc:419] P 6c3ebe1106e84d3f91c76a45bad4bd24: Scheduling UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f): 306516 bytes on disk
I20240502 20:45:56.611508 8846 maintenance_manager.cc:643] P 6c3ebe1106e84d3f91c76a45bad4bd24: UndoDeltaBlockGCOp(68374b60234f4f30ac844bc877056d6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 20:45:56.760443 8724 linked_list-test-util.h:767] Done collecting results (32500 rows in 683.139ms)
I20240502 20:45:56.794304 8724 linked_list-test-util.h:556] Snapshot: P: 1714682735927235 usec, L: 1 verified. Result: OK
I20240502 20:45:56.796998 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 1714682745922732 usec, L: 1 Expected Rows: 63000
I20240502 20:45:56.809600 8724 linked_list-test-util.h:528] Calling callback on tserver 023da3a68b244e89b00248344f835b3a
I20240502 20:45:57.093586 9399 consensus_queue.cc:786] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Peer 5da62881d7084cbf848e7b61285e41a0 is lagging by at least 563 ops behind the committed index [suppressed 6 similar messages]
W20240502 20:45:57.167085 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:45:57.483538 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:57.531646 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:57.551210 8724 linked_list-test-util.h:767] Done collecting results (63000 rows in 741.687ms)
W20240502 20:45:57.562352 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:45:57.599476 8724 linked_list-test-util.h:556] Snapshot: P: 1714682745922732 usec, L: 1 verified. Result: OK
I20240502 20:45:57.601470 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 1714682755900373 usec, L: 1 Expected Rows: 94850
I20240502 20:45:57.606389 8724 linked_list-test-util.h:528] Calling callback on tserver 023da3a68b244e89b00248344f835b3a
W20240502 20:45:57.638803 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:45:58.047922 8842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:45:58.544745 8973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 20:45:58.753577 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:58.798213 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:45:58.906948 8724 linked_list-test-util.h:767] Done collecting results (94850 rows in 1300.63ms)
I20240502 20:45:58.987108 8724 linked_list-test-util.h:556] Snapshot: P: 1714682755900373 usec, L: 1 verified. Result: OK
I20240502 20:45:58.989281 8724 linked_list-test-util.h:509] Verifying Snapshot: NOW Expected Rows: 94850
W20240502 20:45:59.551084 8842 consensus_peers.cc:485] T 273f27ac228d4473ba17b83f75a27ea2 P 6c3ebe1106e84d3f91c76a45bad4bd24 -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:45:59.927492 8973 consensus_peers.cc:485] T 68374b60234f4f30ac844bc877056d6f P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:00.042925 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
W20240502 20:46:00.058650 8973 consensus_peers.cc:485] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a -> Peer 5da62881d7084cbf848e7b61285e41a0 (127.8.133.3:42481): Couldn't send request to peer 5da62881d7084cbf848e7b61285e41a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:00.088810 9264 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:00.367879 8724 linked_list-test-util.h:767] Done collecting results (94850 rows in 1369.66ms)
I20240502 20:46:00.461313 8724 linked_list-test-util.h:556] Snapshot: NOW verified. Result: OK
I20240502 20:46:00.461481 9645 consensus_queue.cc:786] T 2567a0ea66a441138ca0d35f9c477955 P 023da3a68b244e89b00248344f835b3a [LEADER]: Peer 5da62881d7084cbf848e7b61285e41a0 is lagging by at least 563 ops behind the committed index [suppressed 6 similar messages]
I20240502 20:46:00.461913 8724 linked_list-test-util.h:692] Successfully verified 94850 rows
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/linked_list-test.cc:192: Failure
Failed
Bad status: Illegal state: Fewer than 3 TabletServers were alive. Dead TSs:
TabletServer: 5da62881d7084cbf848e7b61285e41a0, Rpc address: host: "127.8.133.3" port: 42481 (Network error: Client connection negotiation failed: client connection to 127.8.133.3:42481: connect: Connection refused (error 111))
I20240502 20:46:00.477761 8724 webui_checker.cc:88] shutting down CURL thread
I20240502 20:46:01.219491 9265 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:01.245172 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 8827
I20240502 20:46:01.258028 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 8958
I20240502 20:46:01.266988 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 8736
2024-05-02T20:46:01Z chronyd exiting
I20240502 20:46:01.284265 8724 test_util.cc:175] -----------------------------------------------
I20240502 20:46:01.284312 8724 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadAndVerify.1714682724761259-8724-0
[ FAILED ] LinkedListTest.TestLoadAndVerify (36513 ms)
[ RUN ] LinkedListTest.TestLoadWhileOneServerDownAndVerify
I20240502 20:46:01.284891 8724 test_util.cc:268] Using random seed: -1327264488
I20240502 20:46:01.285531 8724 linked_list-test.cc:100] Linked List Test Configuration:
I20240502 20:46:01.285573 8724 linked_list-test.cc:101] --------------
I20240502 20:46:01.285583 8724 linked_list-test.cc:102] 50 chains
I20240502 20:46:01.285593 8724 linked_list-test.cc:103] 3 tablets
I20240502 20:46:01.285600 8724 linked_list-test.cc:104] Mutations on
I20240502 20:46:01.285609 8724 linked_list-test.cc:105] --------------
I20240502 20:46:01.285652 8724 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:46:01.285669 8724 ts_itest-base.cc:113] --------------
I20240502 20:46:01.285677 8724 ts_itest-base.cc:114] 3 tablet servers
I20240502 20:46:01.285686 8724 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:46:01.285701 8724 ts_itest-base.cc:116] --------------
2024-05-02T20:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:46:01Z Disabled control of system clock
I20240502 20:46:01.293447 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.133.62:38445
--webserver_interface=127.8.133.62
--webserver_port=0
--builtin_ntp_servers=127.8.133.20:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.133.62:38445
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400 with env {}
W20240502 20:46:01.398636 9658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.398859 9658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:01.398918 9658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.401890 9658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:46:01.401968 9658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:01.401995 9658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:46:01.402019 9658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:46:01.405601 9658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.133.62:38445
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.133.62:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/info.pb
--webserver_interface=127.8.133.62
--webserver_port=0
--tablet_history_max_age_sec=86400
--never_fsync=true
--heap_profile_path=/tmp/kudu.9658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:01.406468 9658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:01.407721 9658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:01.415684 9664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.416155 9663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.417033 9666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:01.417688 9658 server_base.cc:1034] running on GCE node
I20240502 20:46:01.418165 9658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:01.419047 9658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:01.420203 9658 hybrid_clock.cc:648] HybridClock initialized: now 1714682761420188 us; error 27 us; skew 500 ppm
I20240502 20:46:01.429644 9658 webserver.cc:460] Webserver started at http://127.8.133.62:38551/ using document root <none> and password file <none>
I20240502 20:46:01.430229 9658 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:01.430286 9658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:01.430480 9658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:01.432127 9658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/instance:
uuid: "481246e92ed74c15861dba1a5811579b"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.432600 9658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal/instance:
uuid: "481246e92ed74c15861dba1a5811579b"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.435832 9658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 20:46:01.438055 9672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.439041 9658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 20:46:01.439157 9658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
uuid: "481246e92ed74c15861dba1a5811579b"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.439236 9658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:01.453862 9658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:01.454434 9658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:01.454574 9658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:01.461143 9658 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.62:38445
I20240502 20:46:01.461190 9724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.62:38445 every 8 connection(s)
I20240502 20:46:01.462095 9658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/data/info.pb
I20240502 20:46:01.464768 9725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.468036 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 9658
I20240502 20:46:01.468150 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/master-0/wal/instance
I20240502 20:46:01.470137 9725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b: Bootstrap starting.
I20240502 20:46:01.472141 9725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:01.472833 9725 log.cc:824] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:01.474411 9725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b: No bootstrap required, opened a new log
I20240502 20:46:01.477197 9725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481246e92ed74c15861dba1a5811579b" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 38445 } }
I20240502 20:46:01.477389 9725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:01.477444 9725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481246e92ed74c15861dba1a5811579b, State: Initialized, Role: FOLLOWER
I20240502 20:46:01.477839 9725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481246e92ed74c15861dba1a5811579b" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 38445 } }
I20240502 20:46:01.477962 9725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:01.478018 9725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:01.478121 9725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:01.478770 9725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481246e92ed74c15861dba1a5811579b" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 38445 } }
I20240502 20:46:01.479075 9725 leader_election.cc:302] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481246e92ed74c15861dba1a5811579b; no voters:
I20240502 20:46:01.479322 9725 leader_election.cc:288] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 20:46:01.479406 9730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:01.479569 9730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [term 1 LEADER]: Becoming Leader. State: Replica: 481246e92ed74c15861dba1a5811579b, State: Running, Role: LEADER
I20240502 20:46:01.479905 9730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481246e92ed74c15861dba1a5811579b" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 38445 } }
I20240502 20:46:01.480450 9725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:01.481696 9732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481246e92ed74c15861dba1a5811579b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481246e92ed74c15861dba1a5811579b" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 38445 } } }
I20240502 20:46:01.481806 9732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:01.481767 9731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481246e92ed74c15861dba1a5811579b. Latest consensus state: current_term: 1 leader_uuid: "481246e92ed74c15861dba1a5811579b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481246e92ed74c15861dba1a5811579b" member_type: VOTER last_known_addr { host: "127.8.133.62" port: 38445 } } }
I20240502 20:46:01.481878 9731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:01.482697 9734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:01.485698 9734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:01.490774 9734 catalog_manager.cc:1347] Generated new cluster ID: a8fc9a84ba494824b3dc103974af4516
I20240502 20:46:01.490828 9734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:01.500072 9734 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:01.501197 9734 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:01.510072 9734 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 481246e92ed74c15861dba1a5811579b: Generated new TSK 0
I20240502 20:46:01.510754 9734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:01.514406 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.1:0
--local_ip_for_outbound_sockets=127.8.133.1
--webserver_interface=127.8.133.1
--webserver_port=0
--tserver_master_addrs=127.8.133.62:38445
--builtin_ntp_servers=127.8.133.20:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 20:46:01.616693 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.616869 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:01.616935 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.619828 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:01.619943 9749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.1
I20240502 20:46:01.624306 9749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.8.133.1
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.8.133.62:38445
--never_fsync=true
--heap_profile_path=/tmp/kudu.9749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.1
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:01.625273 9749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:01.626610 9749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:01.635564 9755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.636544 9757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.637295 9754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:01.637706 9749 server_base.cc:1034] running on GCE node
I20240502 20:46:01.638142 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:01.638800 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:01.639977 9749 hybrid_clock.cc:648] HybridClock initialized: now 1714682761639955 us; error 32 us; skew 500 ppm
I20240502 20:46:01.642308 9749 webserver.cc:460] Webserver started at http://127.8.133.1:43743/ using document root <none> and password file <none>
I20240502 20:46:01.642985 9749 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:01.643071 9749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:01.643270 9749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:01.645153 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/instance:
uuid: "db7ae73bdf3742ee9a312c9185b70d86"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.645663 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal/instance:
uuid: "db7ae73bdf3742ee9a312c9185b70d86"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.649624 9749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 20:46:01.652387 9763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.653861 9749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 20:46:01.653990 9749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
uuid: "db7ae73bdf3742ee9a312c9185b70d86"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.654103 9749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:01.670055 9749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:01.670583 9749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:01.670701 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:01.671166 9749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:01.672000 9749 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:01.672050 9749 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.672088 9749 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:01.672111 9749 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.680387 9749 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.1:36783
I20240502 20:46:01.680418 9876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.1:36783 every 8 connection(s)
I20240502 20:46:01.681332 9749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
I20240502 20:46:01.690066 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 9749
I20240502 20:46:01.690176 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal/instance
I20240502 20:46:01.690642 9877 heartbeater.cc:347] Connected to a master server at 127.8.133.62:38445
I20240502 20:46:01.690865 9877 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:01.691498 9877 heartbeater.cc:511] Master 127.8.133.62:38445 requested a full tablet report, sending...
I20240502 20:46:01.692306 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.2:0
--local_ip_for_outbound_sockets=127.8.133.2
--webserver_interface=127.8.133.2
--webserver_port=0
--tserver_master_addrs=127.8.133.62:38445
--builtin_ntp_servers=127.8.133.20:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
I20240502 20:46:01.693048 9689 ts_manager.cc:194] Registered new tserver with Master: db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783)
I20240502 20:46:01.694203 9689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.1:37953
W20240502 20:46:01.798460 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.798650 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:01.798702 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.801679 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:01.801792 9880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.2
I20240502 20:46:01.805294 9880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/info.pb
--webserver_interface=127.8.133.2
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.8.133.62:38445
--never_fsync=true
--heap_profile_path=/tmp/kudu.9880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.2
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:01.806104 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:01.807163 9880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:01.815281 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.816090 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.816722 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:01.817456 9880 server_base.cc:1034] running on GCE node
I20240502 20:46:01.817828 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:01.818349 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:01.819499 9880 hybrid_clock.cc:648] HybridClock initialized: now 1714682761819479 us; error 31 us; skew 500 ppm
I20240502 20:46:01.821314 9880 webserver.cc:460] Webserver started at http://127.8.133.2:46871/ using document root <none> and password file <none>
I20240502 20:46:01.821844 9880 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:01.821897 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:01.822086 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:01.823686 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/instance:
uuid: "b35f28ba25a149f6bccfd801d2458095"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.824139 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal/instance:
uuid: "b35f28ba25a149f6bccfd801d2458095"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.827289 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 20:46:01.829465 9894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.830442 9880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 20:46:01.830565 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
uuid: "b35f28ba25a149f6bccfd801d2458095"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:01.830652 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:01.847256 9880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:01.847903 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:01.848055 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:01.848614 9880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:01.849509 9880 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:01.849555 9880 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.849609 9880 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:01.849637 9880 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:01.857728 9880 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.2:43345
I20240502 20:46:01.857769 10007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.2:43345 every 8 connection(s)
I20240502 20:46:01.858628 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/data/info.pb
I20240502 20:46:01.866797 10008 heartbeater.cc:347] Connected to a master server at 127.8.133.62:38445
I20240502 20:46:01.867013 10008 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:01.867507 10008 heartbeater.cc:511] Master 127.8.133.62:38445 requested a full tablet report, sending...
I20240502 20:46:01.867933 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 9880
I20240502 20:46:01.868045 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-1/wal/instance
I20240502 20:46:01.868561 9689 ts_manager.cc:194] Registered new tserver with Master: b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345)
I20240502 20:46:01.869215 9689 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.2:59141
I20240502 20:46:01.870229 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.3:0
--local_ip_for_outbound_sockets=127.8.133.3
--webserver_interface=127.8.133.3
--webserver_port=0
--tserver_master_addrs=127.8.133.62:38445
--builtin_ntp_servers=127.8.133.20:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 20:46:01.974273 10011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:01.974483 10011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:01.974534 10011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:01.977696 10011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:01.977815 10011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.3
I20240502 20:46:01.981472 10011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/info.pb
--webserver_interface=127.8.133.3
--webserver_port=0
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.8.133.62:38445
--never_fsync=true
--heap_profile_path=/tmp/kudu.10011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.3
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:01.982463 10011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:01.983620 10011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:01.992266 10017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.992667 10016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:01.993494 10019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:01.994191 10011 server_base.cc:1034] running on GCE node
I20240502 20:46:01.994540 10011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:01.995060 10011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:01.996204 10011 hybrid_clock.cc:648] HybridClock initialized: now 1714682761996190 us; error 33 us; skew 500 ppm
I20240502 20:46:01.997967 10011 webserver.cc:460] Webserver started at http://127.8.133.3:34377/ using document root <none> and password file <none>
I20240502 20:46:01.998515 10011 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:01.998572 10011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:01.998739 10011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:02.000360 10011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/instance:
uuid: "6ee39e261d9a4c63b3fe9fac09f1a056"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:02.000844 10011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal/instance:
uuid: "6ee39e261d9a4c63b3fe9fac09f1a056"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:02.004045 10011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 20:46:02.006284 10025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:02.007246 10011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 20:46:02.007349 10011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
uuid: "6ee39e261d9a4c63b3fe9fac09f1a056"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:02.007431 10011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:02.045215 10011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:02.045868 10011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:02.046016 10011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:02.046581 10011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:02.047472 10011 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:02.047518 10011 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:02.047571 10011 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:02.047596 10011 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:02.055615 10011 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.3:41435
I20240502 20:46:02.055668 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.3:41435 every 8 connection(s)
I20240502 20:46:02.056500 10011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/data/info.pb
I20240502 20:46:02.064846 10139 heartbeater.cc:347] Connected to a master server at 127.8.133.62:38445
I20240502 20:46:02.065062 10139 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:02.065606 10139 heartbeater.cc:511] Master 127.8.133.62:38445 requested a full tablet report, sending...
I20240502 20:46:02.065707 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 10011
I20240502 20:46:02.065811 8724 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-2/wal/instance
I20240502 20:46:02.066663 9682 ts_manager.cc:194] Registered new tserver with Master: 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
I20240502 20:46:02.067274 9682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.3:60575
I20240502 20:46:02.067891 8724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 20:46:02.076071 9689 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54632:
name: "TestTable"
schema {
columns {
name: "rand_key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "link_to"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "insert_ts"
type: INT64
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "updated"
type: BOOL
is_key: false
is_nullable: false
read_default_value: "\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "rand_key"
}
}
}
W20240502 20:46:02.078689 9689 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 20:46:02.093853 9811 tablet_service.cc:1467] Processing CreateTablet for tablet 188f47b0f1a84de788991856d5239399 (DEFAULT_TABLE table=TestTable [id=5a6f4822d23c44c49d46ce470c0d9e74]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 20:46:02.093883 9942 tablet_service.cc:1467] Processing CreateTablet for tablet 188f47b0f1a84de788991856d5239399 (DEFAULT_TABLE table=TestTable [id=5a6f4822d23c44c49d46ce470c0d9e74]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 20:46:02.093883 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 188f47b0f1a84de788991856d5239399 (DEFAULT_TABLE table=TestTable [id=5a6f4822d23c44c49d46ce470c0d9e74]), partition=RANGE (rand_key) PARTITION UNBOUNDED
I20240502 20:46:02.094708 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188f47b0f1a84de788991856d5239399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.094751 9942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188f47b0f1a84de788991856d5239399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.094751 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188f47b0f1a84de788991856d5239399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:02.099632 10156 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Bootstrap starting.
I20240502 20:46:02.101107 10157 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: Bootstrap starting.
I20240502 20:46:02.102008 10156 tablet_bootstrap.cc:654] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.102810 10156 log.cc:824] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:02.103103 10157 tablet_bootstrap.cc:654] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.103535 10158 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: Bootstrap starting.
I20240502 20:46:02.103838 10157 log.cc:824] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:02.104434 10156 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: No bootstrap required, opened a new log
I20240502 20:46:02.104643 10156 ts_tablet_manager.cc:1367] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240502 20:46:02.105433 10158 tablet_bootstrap.cc:654] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:02.105504 10157 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: No bootstrap required, opened a new log
I20240502 20:46:02.105661 10157 ts_tablet_manager.cc:1367] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:46:02.106161 10158 log.cc:824] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:02.107756 10158 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: No bootstrap required, opened a new log
I20240502 20:46:02.107934 10158 ts_tablet_manager.cc:1367] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 20:46:02.107823 10156 raft_consensus.cc:348] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.108021 10156 raft_consensus.cc:374] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.108074 10156 raft_consensus.cc:729] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7ae73bdf3742ee9a312c9185b70d86, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.108528 10156 consensus_queue.cc:260] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.108901 10157 raft_consensus.cc:348] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.109092 10157 raft_consensus.cc:374] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.109148 10157 raft_consensus.cc:729] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee39e261d9a4c63b3fe9fac09f1a056, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.109318 9877 heartbeater.cc:503] Master 127.8.133.62:38445 was elected leader, sending a full tablet report...
I20240502 20:46:02.109403 10156 ts_tablet_manager.cc:1398] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 20:46:02.109580 10157 consensus_queue.cc:260] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.110214 10139 heartbeater.cc:503] Master 127.8.133.62:38445 was elected leader, sending a full tablet report...
I20240502 20:46:02.110356 10157 ts_tablet_manager.cc:1398] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240502 20:46:02.111390 10158 raft_consensus.cc:348] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.111542 10158 raft_consensus.cc:374] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:02.111583 10158 raft_consensus.cc:729] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35f28ba25a149f6bccfd801d2458095, State: Initialized, Role: FOLLOWER
I20240502 20:46:02.111933 10158 consensus_queue.cc:260] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.112497 10008 heartbeater.cc:503] Master 127.8.133.62:38445 was elected leader, sending a full tablet report...
I20240502 20:46:02.112713 10158 ts_tablet_manager.cc:1398] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 20:46:02.146803 10162 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:02.146991 10162 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.148011 10162 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
I20240502 20:46:02.151477 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188f47b0f1a84de788991856d5239399" candidate_uuid: "db7ae73bdf3742ee9a312c9185b70d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" is_pre_election: true
I20240502 20:46:02.151762 9962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188f47b0f1a84de788991856d5239399" candidate_uuid: "db7ae73bdf3742ee9a312c9185b70d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35f28ba25a149f6bccfd801d2458095" is_pre_election: true
I20240502 20:46:02.151815 10093 raft_consensus.cc:2454] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7ae73bdf3742ee9a312c9185b70d86 in term 0.
I20240502 20:46:02.152030 9962 raft_consensus.cc:2454] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7ae73bdf3742ee9a312c9185b70d86 in term 0.
I20240502 20:46:02.152284 9766 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee39e261d9a4c63b3fe9fac09f1a056, db7ae73bdf3742ee9a312c9185b70d86; no voters:
I20240502 20:46:02.152558 10162 raft_consensus.cc:2781] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:02.152674 10162 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:02.152730 10162 raft_consensus.cc:3037] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.153625 10162 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.153957 10162 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 1 election: Requested vote from peers b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
I20240502 20:46:02.154233 9962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188f47b0f1a84de788991856d5239399" candidate_uuid: "db7ae73bdf3742ee9a312c9185b70d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35f28ba25a149f6bccfd801d2458095"
I20240502 20:46:02.154311 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188f47b0f1a84de788991856d5239399" candidate_uuid: "db7ae73bdf3742ee9a312c9185b70d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056"
I20240502 20:46:02.154351 9962 raft_consensus.cc:3037] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.154423 10093 raft_consensus.cc:3037] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:02.155373 10093 raft_consensus.cc:2454] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7ae73bdf3742ee9a312c9185b70d86 in term 1.
I20240502 20:46:02.155373 9962 raft_consensus.cc:2454] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7ae73bdf3742ee9a312c9185b70d86 in term 1.
I20240502 20:46:02.155655 9766 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee39e261d9a4c63b3fe9fac09f1a056, db7ae73bdf3742ee9a312c9185b70d86; no voters:
I20240502 20:46:02.155808 10162 raft_consensus.cc:2781] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:02.155993 10162 raft_consensus.cc:686] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 1 LEADER]: Becoming Leader. State: Replica: db7ae73bdf3742ee9a312c9185b70d86, State: Running, Role: LEADER
I20240502 20:46:02.156303 10162 consensus_queue.cc:237] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:02.159248 9685 catalog_manager.cc:5484] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1). New cstate: current_term: 1 leader_uuid: "db7ae73bdf3742ee9a312c9185b70d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:02.181258 8724 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240502 20:46:02.182482 9878 tablet.cc:2369] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:02.182811 8724 ts_itest-base.cc:233] Waiting for 1 tablets on tserver db7ae73bdf3742ee9a312c9185b70d86 to finish bootstrapping
I20240502 20:46:02.188033 8724 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b35f28ba25a149f6bccfd801d2458095 to finish bootstrapping
I20240502 20:46:02.192476 8724 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6ee39e261d9a4c63b3fe9fac09f1a056 to finish bootstrapping
I20240502 20:46:02.196784 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 9749
I20240502 20:46:02.205245 8724 linked_list-test-util.h:401] Running for: 30.000s
I20240502 20:46:02.205289 8724 linked_list-test-util.h:402] Sampling every 10000000 us
I20240502 20:46:02.205305 8724 linked_list-test-util.h:408] Written 0 rows in chain
------------------------------------------------------------
Histogram for latency of insert operations (microseconds)
Note: each insert is a batch of 50 rows.
------------------------------------------------------------
Count: 0
Mean: 0
Last: 0
Percentiles:
0% (min) = 0
25% = 0
50% (med) = 0
75% = 0
95% = 0
99% = 0
99.9% = 0
99.99% = 0
100% (max) = 0
W20240502 20:46:02.210009 10142 meta_cache.cc:302] tablet 188f47b0f1a84de788991856d5239399: replica db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783) has failed: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111)
W20240502 20:46:02.218618 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.229072 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.244283 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.253553 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.271708 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.283951 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.307418 10140 tablet.cc:2369] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:46:02.310554 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.325325 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.356058 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.359665 10009 tablet.cc:2369] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:46:02.372525 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.414181 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.434684 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.484030 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.507766 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.557747 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.583582 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.643548 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.673543 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.734755 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.767628 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.837621 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.872351 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.948150 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.987254 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.066629 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.108426 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.193042 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.239887 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.286913 10142 meta_cache.cc:302] tablet 188f47b0f1a84de788991856d5239399: replica db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783) has failed: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111)
W20240502 20:46:03.335449 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.381379 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.478876 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.531742 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.639917 9922 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50988: Illegal state: replica b35f28ba25a149f6bccfd801d2458095 is not leader of this config: current role FOLLOWER
W20240502 20:46:03.693095 10053 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45516: Illegal state: replica 6ee39e261d9a4c63b3fe9fac09f1a056 is not leader of this config: current role FOLLOWER
I20240502 20:46:03.754170 10171 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:03.754316 10171 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:03.755434 10171 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
W20240502 20:46:03.756170 9895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111)
W20240502 20:46:03.757169 9895 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111)
I20240502 20:46:03.759268 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188f47b0f1a84de788991856d5239399" candidate_uuid: "b35f28ba25a149f6bccfd801d2458095" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" is_pre_election: true
I20240502 20:46:03.759416 10093 raft_consensus.cc:2454] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35f28ba25a149f6bccfd801d2458095 in term 1.
I20240502 20:46:03.759738 9897 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ee39e261d9a4c63b3fe9fac09f1a056, b35f28ba25a149f6bccfd801d2458095; no voters: db7ae73bdf3742ee9a312c9185b70d86
I20240502 20:46:03.760017 10171 raft_consensus.cc:2781] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:46:03.760104 10171 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:03.760156 10171 raft_consensus.cc:3037] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:03.761091 10171 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:03.761476 10171 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [CANDIDATE]: Term 2 election: Requested vote from peers db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
I20240502 20:46:03.761893 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188f47b0f1a84de788991856d5239399" candidate_uuid: "b35f28ba25a149f6bccfd801d2458095" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056"
I20240502 20:46:03.762044 10093 raft_consensus.cc:3037] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 1 FOLLOWER]: Advancing to term 2
W20240502 20:46:03.762517 9895 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111)
I20240502 20:46:03.763108 10093 raft_consensus.cc:2454] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35f28ba25a149f6bccfd801d2458095 in term 2.
I20240502 20:46:03.763413 9897 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ee39e261d9a4c63b3fe9fac09f1a056, b35f28ba25a149f6bccfd801d2458095; no voters: db7ae73bdf3742ee9a312c9185b70d86
I20240502 20:46:03.763592 10171 raft_consensus.cc:2781] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:03.763819 10171 raft_consensus.cc:686] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [term 2 LEADER]: Becoming Leader. State: Replica: b35f28ba25a149f6bccfd801d2458095, State: Running, Role: LEADER
I20240502 20:46:03.764197 10171 consensus_queue.cc:237] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:03.767314 9688 catalog_manager.cc:5484] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 reported cstate change: term changed from 1 to 2, leader changed from db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1) to b35f28ba25a149f6bccfd801d2458095 (127.8.133.2). New cstate: current_term: 2 leader_uuid: "b35f28ba25a149f6bccfd801d2458095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:03.804394 10093 raft_consensus.cc:1261] T 188f47b0f1a84de788991856d5239399 P 6ee39e261d9a4c63b3fe9fac09f1a056 [term 2 FOLLOWER]: Refusing update from remote peer b35f28ba25a149f6bccfd801d2458095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 20:46:03.804586 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:03.805126 10176 consensus_queue.cc:1035] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:06.284432 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:07.399660 10188 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 77 ops behind the committed index
W20240502 20:46:09.017246 9895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:46:09.018227 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:10.914917 10188 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 633 ops behind the committed index [suppressed 6 similar messages]
W20240502 20:46:11.327005 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:12.210073 8724 linked_list-test-util.h:417] Sample at HT timestamp: 7023340634960125952 Inserted count: 66000
W20240502 20:46:13.684589 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:14.225304 10212 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 1149 ops behind the committed index [suppressed 6 similar messages]
W20240502 20:46:14.229727 9895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:46:16.414503 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:46:17.605135 10262 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 1691 ops behind the committed index [suppressed 5 similar messages]
W20240502 20:46:19.103817 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:19.586601 9895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 20:46:20.727540 10278 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 2182 ops behind the committed index [suppressed 5 similar messages]
W20240502 20:46:21.858878 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:46:22.211001 8724 linked_list-test-util.h:417] Sample at HT timestamp: 7023340675922522112 Inserted count: 144900
I20240502 20:46:24.114095 10312 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 2704 ops behind the committed index [suppressed 6 similar messages]
W20240502 20:46:24.118412 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:24.660827 9895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:46:26.717823 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:27.175794 10292 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 3179 ops behind the committed index [suppressed 5 similar messages]
W20240502 20:46:29.123278 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:30.081115 9895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 20:46:30.536497 10312 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 3700 ops behind the committed index [suppressed 6 similar messages]
W20240502 20:46:31.487555 9895 consensus_peers.cc:485] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 -> Peer db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783): Couldn't send request to peer db7ae73bdf3742ee9a312c9185b70d86. Status: Network error: Client connection negotiation failed: client connection to 127.8.133.1:36783: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:46:32.205833 8724 linked_list-test-util.h:417] Sample at HT timestamp: 7023340716860874752 Inserted count: 221750
I20240502 20:46:32.205977 8724 linked_list-test-util.h:421] Finished inserting list. Added 221750 in chain
I20240502 20:46:32.206032 8724 linked_list-test-util.h:422] Last entries inserted had keys:
I20240502 20:46:32.206058 8724 linked_list-test-util.h:424] 0: -2700398828671270912
I20240502 20:46:32.206072 8724 linked_list-test-util.h:424] 1: 8298166101575204865
I20240502 20:46:32.206094 8724 linked_list-test-util.h:424] 2: 850127695600418818
I20240502 20:46:32.206108 8724 linked_list-test-util.h:424] 3: -6598051447862657021
I20240502 20:46:32.206136 8724 linked_list-test-util.h:424] 4: 1700114653712547844
I20240502 20:46:32.206153 8724 linked_list-test-util.h:424] 5: -5747923752262238203
I20240502 20:46:32.206173 8724 linked_list-test-util.h:424] 6: 5250641177984237574
I20240502 20:46:32.206192 8724 linked_list-test-util.h:424] 7: -2197397227990548473
I20240502 20:46:32.206209 8724 linked_list-test-util.h:424] 8: 6100768873584656392
I20240502 20:46:32.206229 8724 linked_list-test-util.h:424] 9: -1347410269878419447
I20240502 20:46:32.206249 8724 linked_list-test-util.h:424] 10: -8795167200876494838
I20240502 20:46:32.206266 8724 linked_list-test-util.h:424] 11: 2203397729369980939
I20240502 20:46:32.206287 8724 linked_list-test-util.h:424] 12: -7945180242764365812
I20240502 20:46:32.206306 8724 linked_list-test-util.h:424] 13: 3053525424970399757
I20240502 20:46:32.206322 8724 linked_list-test-util.h:424] 14: -4394653718492676082
I20240502 20:46:32.206342 8724 linked_list-test-util.h:424] 15: 6604051949242089487
I20240502 20:46:32.206362 8724 linked_list-test-util.h:424] 16: -3544526022892257264
I20240502 20:46:32.206378 8724 linked_list-test-util.h:424] 17: 7454038907354218513
I20240502 20:46:32.206398 8724 linked_list-test-util.h:424] 18: 6000501379432466
I20240502 20:46:32.206418 8724 linked_list-test-util.h:424] 19: -7442178642083643373
I20240502 20:46:32.206431 8724 linked_list-test-util.h:424] 20: 856268934468272148
I20240502 20:46:32.206451 8724 linked_list-test-util.h:424] 21: -6591769471506513899
I20240502 20:46:32.206466 8724 linked_list-test-util.h:424] 22: 4406795458739961878
I20240502 20:46:32.206487 8724 linked_list-test-util.h:424] 23: -3041242947234824169
I20240502 20:46:32.206502 8724 linked_list-test-util.h:424] 24: 5256923154340380696
I20240502 20:46:32.206522 8724 linked_list-test-util.h:424] 25: -2191255989122695143
I20240502 20:46:32.206537 8724 linked_list-test-util.h:424] 26: 8807449678612070426
I20240502 20:46:32.206557 8724 linked_list-test-util.h:424] 27: -1341128293522276325
I20240502 20:46:32.206571 8724 linked_list-test-util.h:424] 28: -8789307436985352164
I20240502 20:46:32.206593 8724 linked_list-test-util.h:424] 29: 2209398230749413405
I20240502 20:46:32.206606 8724 linked_list-test-util.h:424] 30: -5238499437736951778
I20240502 20:46:32.206627 8724 linked_list-test-util.h:424] 31: 3059666663838253087
I20240502 20:46:32.206641 8724 linked_list-test-util.h:424] 32: -4388371742136532960
I20240502 20:46:32.206661 8724 linked_list-test-util.h:424] 33: 6610193188109942817
I20240502 20:46:32.206676 8724 linked_list-test-util.h:424] 34: -837845217864843230
I20240502 20:46:32.206696 8724 linked_list-test-util.h:424] 35: 7460320883710361635
I20240502 20:46:32.206710 8724 linked_list-test-util.h:424] 36: 12141740247285796
I20240502 20:46:32.206730 8724 linked_list-test-util.h:424] 37: -7435896665727500251
I20240502 20:46:32.206744 8724 linked_list-test-util.h:424] 38: 3562668264518975526
I20240502 20:46:32.206764 8724 linked_list-test-util.h:424] 39: -6585909707615371225
I20240502 20:46:32.206779 8724 linked_list-test-util.h:424] 40: 4413077435096105000
I20240502 20:46:32.206799 8724 linked_list-test-util.h:424] 41: -3035101708366970839
I20240502 20:46:32.206815 8724 linked_list-test-util.h:424] 42: 7963603959367794730
I20240502 20:46:32.206835 8724 linked_list-test-util.h:424] 43: -2184974012766552021
I20240502 20:46:32.206849 8724 linked_list-test-util.h:424] 44: 8813590917479923756
I20240502 20:46:32.206869 8724 linked_list-test-util.h:424] 45: 1365552511505137709
I20240502 20:46:32.206884 8724 linked_list-test-util.h:424] 46: -6082626631957938130
I20240502 20:46:32.206905 8724 linked_list-test-util.h:424] 47: 2215539469617266735
I20240502 20:46:32.206919 8724 linked_list-test-util.h:424] 48: -5232498936357519312
I20240502 20:46:32.206939 8724 linked_list-test-util.h:424] 49: 5766065993888956465
I20240502 20:46:32.306623 8724 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
/tmp/dist-test-taskRRMtjR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.133.1:36783
--local_ip_for_outbound_sockets=127.8.133.1
--tserver_master_addrs=127.8.133.62:38445
--webserver_port=43743
--webserver_interface=127.8.133.1
--builtin_ntp_servers=127.8.133.20:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir
--tablet_history_max_age_sec=86400
--log_cache_size_limit_mb=2
--global_log_cache_size_limit_mb=4 with env {}
W20240502 20:46:32.421530 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:32.421718 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:32.421764 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:32.424789 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:32.424893 10368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.133.1
I20240502 20:46:32.428306 10368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.133.20:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--skip_remove_old_recovery_dir=true
--global_log_cache_size_limit_mb=4
--log_cache_size_limit_mb=2
--fs_data_dirs=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.133.1:36783
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
--webserver_interface=127.8.133.1
--webserver_port=43743
--tablet_history_max_age_sec=86400
--tserver_master_addrs=127.8.133.62:38445
--never_fsync=true
--heap_profile_path=/tmp/kudu.10368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.133.1
--log_dir=/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:32.429107 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:32.430159 10368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:32.438273 10374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:32.438879 10368 server_base.cc:1034] running on GCE node
W20240502 20:46:32.439306 10373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:32.439738 10376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:32.440145 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:32.440687 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:32.441866 10368 hybrid_clock.cc:648] HybridClock initialized: now 1714682792441845 us; error 29 us; skew 500 ppm
I20240502 20:46:32.443771 10368 webserver.cc:460] Webserver started at http://127.8.133.1:43743/ using document root <none> and password file <none>
I20240502 20:46:32.444307 10368 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:32.444376 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:32.447584 10368 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 20:46:32.449489 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 20:46:32.450435 10368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 20:46:32.450537 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data,/tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
uuid: "db7ae73bdf3742ee9a312c9185b70d86"
format_stamp: "Formatted at 2024-05-02 20:46:01 on dist-test-slave-3019"
I20240502 20:46:32.450917 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:32.474220 10368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 20:46:32.474936 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:32.475103 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:32.475680 10368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:32.476873 10389 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 20:46:32.480198 10368 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 20:46:32.480284 10368 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240502 20:46:32.480369 10368 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 20:46:32.482170 10368 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 20:46:32.482228 10368 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240502 20:46:32.482486 10389 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Bootstrap starting.
I20240502 20:46:32.492790 10368 rpc_server.cc:307] RPC server started. Bound to: 127.8.133.1:36783
I20240502 20:46:32.493855 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/data/info.pb
I20240502 20:46:32.494606 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.133.1:36783 every 8 connection(s)
I20240502 20:46:32.497524 10389 log.cc:824] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:32.500984 10389 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:46:32.501236 10389 log.cc:1225] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: --skip_remove_old_recovery_dir enabled. NOT deleting /tmp/dist-test-taskRRMtjR/test-tmp/linked_list-test.0.LinkedListTest.TestLoadWhileOneServerDownAndVerify.1714682724761259-8724-0/linked-list-cluster/ts-0/wal/wals/188f47b0f1a84de788991856d5239399.recovery-1714682792501181
I20240502 20:46:32.501323 10389 tablet_bootstrap.cc:492] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Bootstrap complete.
I20240502 20:46:32.501580 10389 ts_tablet_manager.cc:1367] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20240502 20:46:32.502885 10497 heartbeater.cc:347] Connected to a master server at 127.8.133.62:38445
I20240502 20:46:32.503104 10497 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:32.503387 8724 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu as pid 10368
I20240502 20:46:32.503650 10497 heartbeater.cc:511] Master 127.8.133.62:38445 requested a full tablet report, sending...
I20240502 20:46:32.505798 9682 ts_manager.cc:194] Re-registered known tserver with Master: db7ae73bdf3742ee9a312c9185b70d86 (127.8.133.1:36783)
I20240502 20:46:32.506724 9682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.133.1:35287
I20240502 20:46:32.509092 10389 raft_consensus.cc:348] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:32.509721 10389 raft_consensus.cc:729] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7ae73bdf3742ee9a312c9185b70d86, State: Initialized, Role: FOLLOWER
I20240502 20:46:32.510296 10389 consensus_queue.cc:260] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:32.511333 10389 ts_tablet_manager.cc:1398] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20240502 20:46:32.511507 10497 heartbeater.cc:503] Master 127.8.133.62:38445 was elected leader, sending a full tablet report...
I20240502 20:46:32.518767 10451 raft_consensus.cc:3037] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:32.520802 10451 pending_rounds.cc:76] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Aborting all ops after (but not including) 0
I20240502 20:46:32.520852 10451 pending_rounds.cc:97] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 20:46:32.520928 10451 raft_consensus.cc:2866] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 20:46:32.566699 8724 cluster_itest_util.cc:256] Not converged past 4435 yet: 2.4464 2.4464 2.500
I20240502 20:46:32.719719 8724 cluster_itest_util.cc:256] Not converged past 4435 yet: 2.4464 2.4464 2.984
I20240502 20:46:32.922185 8724 cluster_itest_util.cc:256] Not converged past 4435 yet: 2.4464 2.4464 2.1467
I20240502 20:46:33.029033 10508 mvcc.cc:205] Tried to move back new op lower bound from 7023340613292118016 to 7023340600380907520. Current Snapshot: MvccSnapshot[applied={T|T < 7023340613292118016}]
I20240502 20:46:33.224674 8724 cluster_itest_util.cc:256] Not converged past 4435 yet: 2.4464 2.4464 2.2443
I20240502 20:46:33.589955 10339 consensus_queue.cc:786] T 188f47b0f1a84de788991856d5239399 P b35f28ba25a149f6bccfd801d2458095 [LEADER]: Peer db7ae73bdf3742ee9a312c9185b70d86 is lagging by at least 89 ops behind the committed index [suppressed 11 similar messages]
I20240502 20:46:33.685252 8724 cluster_itest_util.cc:256] Not converged past 4435 yet: 2.4464 2.4464 2.4375
I20240502 20:46:34.187675 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 9880
I20240502 20:46:34.202113 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 10011
I20240502 20:46:34.216753 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 1714682772207062 usec, L: 1 Expected Rows: 66000
I20240502 20:46:34.218106 8724 meta_cache.cc:1510] marking tablet server b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345) as failed
W20240502 20:46:34.218204 8724 meta_cache.cc:302] tablet 188f47b0f1a84de788991856d5239399: replica b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.8.133.2:43345: connect: Connection refused (error 111)
I20240502 20:46:34.242604 8724 linked_list-test-util.h:528] Calling callback on tserver db7ae73bdf3742ee9a312c9185b70d86
I20240502 20:46:35.633327 8724 linked_list-test-util.h:767] Done collecting results (66000 rows in 1390.81ms)
I20240502 20:46:35.676384 10517 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b35f28ba25a149f6bccfd801d2458095)
I20240502 20:46:35.676594 10517 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:35.677789 10517 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
W20240502 20:46:35.678426 10384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.2:43345: connect: Connection refused (error 111)
W20240502 20:46:35.679311 10384 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345): Network error: Client connection negotiation failed: client connection to 127.8.133.2:43345: connect: Connection refused (error 111)
W20240502 20:46:35.679417 10385 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435): Network error: Client connection negotiation failed: client connection to 127.8.133.3:41435: connect: Connection refused (error 111)
I20240502 20:46:35.679503 10385 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7ae73bdf3742ee9a312c9185b70d86; no voters: 6ee39e261d9a4c63b3fe9fac09f1a056, b35f28ba25a149f6bccfd801d2458095
I20240502 20:46:35.679692 10517 raft_consensus.cc:2726] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:46:35.689159 8724 linked_list-test-util.h:556] Snapshot: P: 1714682772207062 usec, L: 1 verified. Result: OK
I20240502 20:46:35.701314 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 1714682782207647 usec, L: 1 Expected Rows: 144900
I20240502 20:46:35.710548 8724 linked_list-test-util.h:528] Calling callback on tserver db7ae73bdf3742ee9a312c9185b70d86
I20240502 20:46:37.531157 10521 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b35f28ba25a149f6bccfd801d2458095)
I20240502 20:46:37.531394 10521 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:37.531865 10521 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
W20240502 20:46:37.533036 10385 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435): Network error: Client connection negotiation failed: client connection to 127.8.133.3:41435: connect: Connection refused (error 111)
W20240502 20:46:37.533275 10384 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345): Network error: Client connection negotiation failed: client connection to 127.8.133.2:43345: connect: Connection refused (error 111)
I20240502 20:46:37.533352 10384 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7ae73bdf3742ee9a312c9185b70d86; no voters: 6ee39e261d9a4c63b3fe9fac09f1a056, b35f28ba25a149f6bccfd801d2458095
I20240502 20:46:37.533499 10521 raft_consensus.cc:2726] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:46:37.934549 8724 linked_list-test-util.h:767] Done collecting results (144900 rows in 2224.08ms)
I20240502 20:46:38.040220 8724 linked_list-test-util.h:556] Snapshot: P: 1714682782207647 usec, L: 1 verified. Result: OK
I20240502 20:46:38.042430 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 1714682792202362 usec, L: 1 Expected Rows: 221750
I20240502 20:46:38.047701 8724 linked_list-test-util.h:528] Calling callback on tserver db7ae73bdf3742ee9a312c9185b70d86
I20240502 20:46:40.059765 8724 linked_list-test-util.h:767] Done collecting results (221750 rows in 2012.14ms)
I20240502 20:46:40.230490 8724 linked_list-test-util.h:556] Snapshot: P: 1714682792202362 usec, L: 1 verified. Result: OK
I20240502 20:46:40.232664 8724 linked_list-test-util.h:509] Verifying Snapshot: P: 4503599627370495 usec, L: 4094 Expected Rows: 221750
I20240502 20:46:40.234573 8724 linked_list-test-util.h:528] Calling callback on tserver db7ae73bdf3742ee9a312c9185b70d86
I20240502 20:46:40.743227 10525 raft_consensus.cc:482] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b35f28ba25a149f6bccfd801d2458095)
I20240502 20:46:40.743379 10525 raft_consensus.cc:504] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7ae73bdf3742ee9a312c9185b70d86" member_type: VOTER last_known_addr { host: "127.8.133.1" port: 36783 } } peers { permanent_uuid: "b35f28ba25a149f6bccfd801d2458095" member_type: VOTER last_known_addr { host: "127.8.133.2" port: 43345 } } peers { permanent_uuid: "6ee39e261d9a4c63b3fe9fac09f1a056" member_type: VOTER last_known_addr { host: "127.8.133.3" port: 41435 } }
I20240502 20:46:40.743834 10525 leader_election.cc:288] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345), 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435)
W20240502 20:46:40.744436 10384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.133.2:43345: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 20:46:40.745337 10385 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ee39e261d9a4c63b3fe9fac09f1a056 (127.8.133.3:41435): Network error: Client connection negotiation failed: client connection to 127.8.133.3:41435: connect: Connection refused (error 111)
W20240502 20:46:40.745440 10384 leader_election.cc:334] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b35f28ba25a149f6bccfd801d2458095 (127.8.133.2:43345): Network error: Client connection negotiation failed: client connection to 127.8.133.2:43345: connect: Connection refused (error 111)
I20240502 20:46:40.745529 10384 leader_election.cc:302] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7ae73bdf3742ee9a312c9185b70d86; no voters: 6ee39e261d9a4c63b3fe9fac09f1a056, b35f28ba25a149f6bccfd801d2458095
I20240502 20:46:40.745679 10525 raft_consensus.cc:2726] T 188f47b0f1a84de788991856d5239399 P db7ae73bdf3742ee9a312c9185b70d86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:46:42.295825 8724 linked_list-test-util.h:767] Done collecting results (221750 rows in 2061.33ms)
I20240502 20:46:42.465476 8724 linked_list-test-util.h:556] Snapshot: P: 4503599627370495 usec, L: 4094 verified. Result: OK
I20240502 20:46:42.465818 8724 linked_list-test-util.h:692] Successfully verified 221750 rows
I20240502 20:46:42.466346 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 10368
W20240502 20:46:42.467150 10142 client.cc:2025] Couldn't close scanner 75db11b5d0194aceafd4d7583393ec4a: Service unavailable: reactor is shutting down (error 108)
I20240502 20:46:42.478099 8724 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRRMtjR/build/debug/bin/kudu with pid 9658
2024-05-02T20:46:42Z chronyd exiting
[ OK ] LinkedListTest.TestLoadWhileOneServerDownAndVerify (41218 ms)
[----------] 2 tests from LinkedListTest (77731 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (77732 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] LinkedListTest.TestLoadAndVerify
1 FAILED TEST