Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 20:07:16.881582 21243 test_util.cc:276] Using random seed: 464615586
I20250626 20:07:16.886054 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:16.886093 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:16.886113 21243 ts_itest-base.cc:117] 4 tablet servers
I20250626 20:07:16.886134 21243 ts_itest-base.cc:118] 3 replicas per TS
I20250626 20:07:16.886152 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:16Z Disabled control of system clock
I20250626 20:07:16.892374 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:35577
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:35577 with env {}
W20250626 20:07:16.958071 21255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:16.958185 21255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:16.958201 21255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:16.959348 21255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:16.959384 21255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:16.959396 21255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:16.959406 21255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:16.960595 21255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:35577
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:35577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:16.960747 21255 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:16.960918 21255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:16.964213 21261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:16.964234 21263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:16.964293 21260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:16.964376 21255 server_base.cc:1048] running on GCE node
I20250626 20:07:16.964584 21255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:16.964773 21255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:16.965899 21255 hybrid_clock.cc:648] HybridClock initialized: now 1750968436965871 us; error 39 us; skew 500 ppm
I20250626 20:07:16.966959 21255 webserver.cc:469] Webserver started at http://127.20.190.254:40031/ using document root <none> and password file <none>
I20250626 20:07:16.967110 21255 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:16.967149 21255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:16.967238 21255 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:16.967880 21255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "75a436c503914738b9c08c867641edde"
format_stamp: "Formatted at 2025-06-26 20:07:16 on dist-test-slave-l47t"
I20250626 20:07:16.968119 21255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "75a436c503914738b9c08c867641edde"
format_stamp: "Formatted at 2025-06-26 20:07:16 on dist-test-slave-l47t"
I20250626 20:07:16.969134 21255 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:16.969641 21269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:16.969785 21255 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:16.969846 21255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "75a436c503914738b9c08c867641edde"
format_stamp: "Formatted at 2025-06-26 20:07:16 on dist-test-slave-l47t"
I20250626 20:07:16.969893 21255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:16.987392 21255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:16.987627 21255 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:16.987737 21255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:16.991073 21255 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:35577
I20250626 20:07:16.991107 21321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:35577 every 8 connection(s)
I20250626 20:07:16.991346 21255 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:16.991827 21322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:16.993852 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Bootstrap starting.
I20250626 20:07:16.994316 21322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:16.994501 21322 log.cc:826] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:16.994974 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: No bootstrap required, opened a new log
I20250626 20:07:16.995385 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 21255
I20250626 20:07:16.995474 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:16.996085 21322 raft_consensus.cc:357] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:16.996193 21322 raft_consensus.cc:383] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:16.996228 21322 raft_consensus.cc:738] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a436c503914738b9c08c867641edde, State: Initialized, Role: FOLLOWER
I20250626 20:07:16.996383 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:16.996443 21322 raft_consensus.cc:397] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:16.996505 21322 raft_consensus.cc:491] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:16.996572 21322 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:16.997088 21322 raft_consensus.cc:513] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:16.997184 21322 leader_election.cc:304] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a436c503914738b9c08c867641edde; no voters:
I20250626 20:07:16.997308 21322 leader_election.cc:290] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:16.997364 21327 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:16.997489 21322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:16.997537 21327 raft_consensus.cc:695] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 LEADER]: Becoming Leader. State: Replica: 75a436c503914738b9c08c867641edde, State: Running, Role: LEADER
I20250626 20:07:16.997668 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:16.997953 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a436c503914738b9c08c867641edde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } } }
I20250626 20:07:16.998080 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:16.998247 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a436c503914738b9c08c867641edde. Latest consensus state: current_term: 1 leader_uuid: "75a436c503914738b9c08c867641edde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } } }
I20250626 20:07:16.998348 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:16.998500 21336 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:16.998953 21336 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:17.000344 21336 catalog_manager.cc:1349] Generated new cluster ID: 8e840a7722d44d22b6276feb12ff9376
I20250626 20:07:17.000396 21336 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:17.020741 21336 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:17.021162 21336 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:17.025354 21336 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Generated new TSK 0
I20250626 20:07:17.025509 21336 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:17.033067 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--builtin_ntp_servers=127.20.190.212:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:17.098656 21347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:17.098803 21347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:17.098829 21347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:17.100040 21347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:17.100085 21347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:17.101272 21347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--never_fsync=true
--heap_profile_path=/tmp/kudu.21347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:17.101469 21347 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:17.101656 21347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:17.104600 21353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.104612 21352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.104709 21355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.105440 21347 server_base.cc:1048] running on GCE node
I20250626 20:07:17.105588 21347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:17.105764 21347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:17.106882 21347 hybrid_clock.cc:648] HybridClock initialized: now 1750968437106869 us; error 31 us; skew 500 ppm
I20250626 20:07:17.107932 21347 webserver.cc:469] Webserver started at http://127.20.190.193:33091/ using document root <none> and password file <none>
I20250626 20:07:17.108083 21347 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:17.108120 21347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:17.108209 21347 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:17.108860 21347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "be51f2f7007e443c8772aeec3964caa0"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.109098 21347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "be51f2f7007e443c8772aeec3964caa0"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.110141 21347 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:17.110762 21361 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.110908 21347 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:17.110972 21347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "be51f2f7007e443c8772aeec3964caa0"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.111034 21347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:17.122331 21347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:17.122586 21347 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:17.122692 21347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:17.122869 21347 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:17.123101 21347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:17.123129 21347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.123158 21347 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:17.123188 21347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.128257 21347 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:46823
I20250626 20:07:17.128315 21474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:46823 every 8 connection(s)
I20250626 20:07:17.128576 21347 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:17.132182 21475 heartbeater.cc:344] Connected to a master server at 127.20.190.254:35577
I20250626 20:07:17.132251 21475 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:17.132368 21475 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:17.132730 21286 ts_manager.cc:194] Registered new tserver with Master: be51f2f7007e443c8772aeec3964caa0 (127.20.190.193:46823)
I20250626 20:07:17.133316 21286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:40261
I20250626 20:07:17.136590 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 21347
I20250626 20:07:17.136646 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:17.137406 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--builtin_ntp_servers=127.20.190.212:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:17.201325 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:17.201447 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:17.201463 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:17.202548 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:17.202584 21478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:17.203713 21478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:17.203857 21478 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:17.203996 21478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:17.207151 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.207176 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.207170 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.207293 21478 server_base.cc:1048] running on GCE node
I20250626 20:07:17.207463 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:17.207600 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:17.208719 21478 hybrid_clock.cc:648] HybridClock initialized: now 1750968437208708 us; error 22 us; skew 500 ppm
I20250626 20:07:17.209650 21478 webserver.cc:469] Webserver started at http://127.20.190.194:44797/ using document root <none> and password file <none>
I20250626 20:07:17.209806 21478 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:17.209843 21478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:17.209937 21478 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:17.210794 21478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.211097 21478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.213766 21478 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250626 20:07:17.214339 21492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.214473 21478 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:17.214548 21478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.214601 21478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:17.231007 21478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:17.231210 21478 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:17.231307 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:17.231470 21478 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:17.231704 21478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:17.231734 21478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.231761 21478 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:17.231779 21478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.236342 21478 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:39351
I20250626 20:07:17.236423 21605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:39351 every 8 connection(s)
I20250626 20:07:17.236632 21478 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:17.239755 21606 heartbeater.cc:344] Connected to a master server at 127.20.190.254:35577
I20250626 20:07:17.239828 21606 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:17.239982 21606 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:17.240231 21286 ts_manager.cc:194] Registered new tserver with Master: 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194:39351)
I20250626 20:07:17.240530 21286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:44479
I20250626 20:07:17.240782 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 21478
I20250626 20:07:17.240849 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 20:07:17.241602 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--builtin_ntp_servers=127.20.190.212:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:17.306599 21609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:17.306718 21609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:17.306733 21609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:17.307859 21609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:17.307904 21609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:17.309087 21609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--never_fsync=true
--heap_profile_path=/tmp/kudu.21609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:17.309252 21609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:17.309427 21609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:17.312335 21615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.312356 21617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.312520 21609 server_base.cc:1048] running on GCE node
W20250626 20:07:17.312350 21614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.312726 21609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:17.312858 21609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:17.313967 21609 hybrid_clock.cc:648] HybridClock initialized: now 1750968437313960 us; error 23 us; skew 500 ppm
I20250626 20:07:17.314932 21609 webserver.cc:469] Webserver started at http://127.20.190.195:42265/ using document root <none> and password file <none>
I20250626 20:07:17.315078 21609 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:17.315110 21609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:17.315232 21609 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:17.315961 21609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a1b9045c2112415fa364df635e08f0c8"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.316223 21609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a1b9045c2112415fa364df635e08f0c8"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.317173 21609 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:17.317700 21623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.317855 21609 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:17.317909 21609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a1b9045c2112415fa364df635e08f0c8"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.317958 21609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:17.332942 21609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:17.333135 21609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:17.333225 21609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:17.333376 21609 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:17.333603 21609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:17.333631 21609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.333657 21609 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:17.333683 21609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.338408 21609 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:36103
I20250626 20:07:17.338469 21736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:36103 every 8 connection(s)
I20250626 20:07:17.338676 21609 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:17.341845 21737 heartbeater.cc:344] Connected to a master server at 127.20.190.254:35577
I20250626 20:07:17.341909 21737 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:17.342056 21737 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:17.342334 21286 ts_manager.cc:194] Registered new tserver with Master: a1b9045c2112415fa364df635e08f0c8 (127.20.190.195:36103)
I20250626 20:07:17.342721 21286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:33797
I20250626 20:07:17.344728 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 21609
I20250626 20:07:17.344787 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 20:07:17.345618 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.196:0
--local_ip_for_outbound_sockets=127.20.190.196
--webserver_interface=127.20.190.196
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--builtin_ntp_servers=127.20.190.212:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:17.410352 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:17.410478 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:17.410494 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:17.411631 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:17.411677 21740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.196
I20250626 20:07:17.412865 21740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.190.196
--webserver_port=0
--tserver_master_addrs=127.20.190.254:35577
--never_fsync=true
--heap_profile_path=/tmp/kudu.21740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.196
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:17.413002 21740 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:17.413177 21740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:17.416086 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.416188 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.416383 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.416394 21740 server_base.cc:1048] running on GCE node
I20250626 20:07:17.416554 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:17.416690 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:17.417797 21740 hybrid_clock.cc:648] HybridClock initialized: now 1750968437417783 us; error 20 us; skew 500 ppm
I20250626 20:07:17.418800 21740 webserver.cc:469] Webserver started at http://127.20.190.196:44729/ using document root <none> and password file <none>
I20250626 20:07:17.418951 21740 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:17.418982 21740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:17.419067 21740 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:17.419708 21740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.419941 21740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.420886 21740 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:17.421411 21754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.421540 21740 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:17.421593 21740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
format_stamp: "Formatted at 2025-06-26 20:07:17 on dist-test-slave-l47t"
I20250626 20:07:17.421651 21740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:17.431612 21740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:17.431813 21740 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:17.431901 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:17.432068 21740 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:17.432287 21740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:17.432313 21740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.432339 21740 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:17.432371 21740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.437070 21740 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.196:41229
I20250626 20:07:17.437125 21867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.196:41229 every 8 connection(s)
I20250626 20:07:17.437312 21740 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 20:07:17.438923 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 21740
I20250626 20:07:17.438987 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 20:07:17.440742 21868 heartbeater.cc:344] Connected to a master server at 127.20.190.254:35577
I20250626 20:07:17.440809 21868 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:17.440975 21868 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:17.441238 21286 ts_manager.cc:194] Registered new tserver with Master: 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196:41229)
I20250626 20:07:17.441530 21286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.196:43413
I20250626 20:07:17.450634 21243 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 20:07:17.457945 21286 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36022:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 20:07:17.464135 21540 tablet_service.cc:1468] Processing CreateTablet for tablet e9fd9eaebc174277ae2d8e763fe832b6 (DEFAULT_TABLE table=TestTable [id=f34202be713b4f4d8c6c59f81b2adc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:17.464367 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fd9eaebc174277ae2d8e763fe832b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:17.464565 21802 tablet_service.cc:1468] Processing CreateTablet for tablet e9fd9eaebc174277ae2d8e763fe832b6 (DEFAULT_TABLE table=TestTable [id=f34202be713b4f4d8c6c59f81b2adc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:17.464643 21671 tablet_service.cc:1468] Processing CreateTablet for tablet e9fd9eaebc174277ae2d8e763fe832b6 (DEFAULT_TABLE table=TestTable [id=f34202be713b4f4d8c6c59f81b2adc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:17.464771 21802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fd9eaebc174277ae2d8e763fe832b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:17.464876 21671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fd9eaebc174277ae2d8e763fe832b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:17.466622 21882 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: Bootstrap starting.
I20250626 20:07:17.466832 21883 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: Bootstrap starting.
I20250626 20:07:17.467142 21882 tablet_bootstrap.cc:654] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:17.467350 21884 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Bootstrap starting.
I20250626 20:07:17.467396 21882 log.cc:826] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:17.467409 21883 tablet_bootstrap.cc:654] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:17.467664 21883 log.cc:826] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:17.467870 21884 tablet_bootstrap.cc:654] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:17.467978 21882 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: No bootstrap required, opened a new log
I20250626 20:07:17.468042 21882 ts_tablet_manager.cc:1397] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:17.468114 21884 log.cc:826] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:17.468178 21883 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: No bootstrap required, opened a new log
I20250626 20:07:17.468230 21883 ts_tablet_manager.cc:1397] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:17.468602 21884 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: No bootstrap required, opened a new log
I20250626 20:07:17.468654 21884 ts_tablet_manager.cc:1397] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:17.469226 21883 raft_consensus.cc:357] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.469306 21883 raft_consensus.cc:383] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:17.469322 21883 raft_consensus.cc:738] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1dd20e1a5d4542b74bc1e25b033ce7, State: Initialized, Role: FOLLOWER
I20250626 20:07:17.469385 21883 consensus_queue.cc:260] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.469485 21882 raft_consensus.cc:357] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.469569 21883 ts_tablet_manager.cc:1428] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:17.469586 21882 raft_consensus.cc:383] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:17.469611 21882 raft_consensus.cc:738] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0854b1ce3f074bf580eefc095bd3e6cc, State: Initialized, Role: FOLLOWER
I20250626 20:07:17.469681 21882 consensus_queue.cc:260] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.469866 21882 ts_tablet_manager.cc:1428] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:17.469880 21868 heartbeater.cc:499] Master 127.20.190.254:35577 was elected leader, sending a full tablet report...
I20250626 20:07:17.469877 21884 raft_consensus.cc:357] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.469966 21884 raft_consensus.cc:383] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:17.469985 21884 raft_consensus.cc:738] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b9045c2112415fa364df635e08f0c8, State: Initialized, Role: FOLLOWER
I20250626 20:07:17.470041 21884 consensus_queue.cc:260] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.470214 21884 ts_tablet_manager.cc:1428] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:17.470239 21737 heartbeater.cc:499] Master 127.20.190.254:35577 was elected leader, sending a full tablet report...
I20250626 20:07:17.470297 21606 heartbeater.cc:499] Master 127.20.190.254:35577 was elected leader, sending a full tablet report...
I20250626 20:07:17.481887 21888 raft_consensus.cc:491] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:17.481940 21888 raft_consensus.cc:513] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.482136 21888 leader_election.cc:290] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196:41229), a1b9045c2112415fa364df635e08f0c8 (127.20.190.195:36103)
I20250626 20:07:17.484448 21822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" is_pre_election: true
I20250626 20:07:17.484531 21822 raft_consensus.cc:2466] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1dd20e1a5d4542b74bc1e25b033ce7 in term 0.
I20250626 20:07:17.484704 21493 leader_election.cc:304] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0854b1ce3f074bf580eefc095bd3e6cc, 2b1dd20e1a5d4542b74bc1e25b033ce7; no voters:
I20250626 20:07:17.484714 21691 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9045c2112415fa364df635e08f0c8" is_pre_election: true
I20250626 20:07:17.484799 21691 raft_consensus.cc:2466] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1dd20e1a5d4542b74bc1e25b033ce7 in term 0.
I20250626 20:07:17.484879 21888 raft_consensus.cc:2802] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:17.484944 21888 raft_consensus.cc:491] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:17.484975 21888 raft_consensus.cc:3058] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:17.485481 21888 raft_consensus.cc:513] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:17.485601 21888 leader_election.cc:290] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196:41229), a1b9045c2112415fa364df635e08f0c8 (127.20.190.195:36103)
I20250626 20:07:17.485741 21691 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b9045c2112415fa364df635e08f0c8"
I20250626 20:07:17.485745 21822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
I20250626 20:07:17.485798 21822 raft_consensus.cc:3058] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:17.485803 21691 raft_consensus.cc:3058] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:17.486343 21822 raft_consensus.cc:2466] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1dd20e1a5d4542b74bc1e25b033ce7 in term 1.
I20250626 20:07:17.486364 21691 raft_consensus.cc:2466] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1dd20e1a5d4542b74bc1e25b033ce7 in term 1.
I20250626 20:07:17.486590 21494 leader_election.cc:304] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1dd20e1a5d4542b74bc1e25b033ce7, a1b9045c2112415fa364df635e08f0c8; no voters:
I20250626 20:07:17.486676 21888 raft_consensus.cc:2802] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:17.486799 21888 raft_consensus.cc:695] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 2b1dd20e1a5d4542b74bc1e25b033ce7, State: Running, Role: LEADER
I20250626 20:07:17.486879 21888 consensus_queue.cc:237] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
W20250626 20:07:17.487035 21607 tablet.cc:2378] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:17.487484 21285 catalog_manager.cc:5582] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194). New cstate: current_term: 1 leader_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:17.497107 21243 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 20:07:17.497493 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2b1dd20e1a5d4542b74bc1e25b033ce7 to finish bootstrapping
I20250626 20:07:17.500217 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a1b9045c2112415fa364df635e08f0c8 to finish bootstrapping
I20250626 20:07:17.502367 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0854b1ce3f074bf580eefc095bd3e6cc to finish bootstrapping
I20250626 20:07:17.504566 21243 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250626 20:07:17.505004 21243 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
rpc_addresses {
host: "127.20.190.196"
port: 41229
}
}
ts_infos {
permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7"
rpc_addresses {
host: "127.20.190.194"
port: 39351
}
}
ts_infos {
permanent_uuid: "a1b9045c2112415fa364df635e08f0c8"
rpc_addresses {
host: "127.20.190.195"
port: 36103
}
}
W20250626 20:07:17.589145 21738 tablet.cc:2378] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:17.687698 21869 tablet.cc:2378] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:17.871528 21888 consensus_queue.cc:1035] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:17.873256 21897 consensus_queue.cc:1035] T e9fd9eaebc174277ae2d8e763fe832b6 P 2b1dd20e1a5d4542b74bc1e25b033ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:17.878172 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 21478
I20250626 20:07:17.883280 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 21255
I20250626 20:07:17.889384 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:35577
--webserver_interface=127.20.190.254
--webserver_port=40031
--builtin_ntp_servers=127.20.190.212:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:35577 with env {}
W20250626 20:07:17.956331 21901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:17.956485 21901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:17.956501 21901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:17.957618 21901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:17.957650 21901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:17.957661 21901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:17.957672 21901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:17.958894 21901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:35577
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:35577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=40031
--never_fsync=true
--heap_profile_path=/tmp/kudu.21901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:17.959033 21901 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:17.959179 21901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:17.961905 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:17.961930 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.962041 21901 server_base.cc:1048] running on GCE node
W20250626 20:07:17.962127 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:17.962328 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:17.962541 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:17.963665 21901 hybrid_clock.cc:648] HybridClock initialized: now 1750968437963653 us; error 27 us; skew 500 ppm
I20250626 20:07:17.964644 21901 webserver.cc:469] Webserver started at http://127.20.190.254:40031/ using document root <none> and password file <none>
I20250626 20:07:17.964814 21901 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:17.964871 21901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:17.965932 21901 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 20:07:17.966472 21915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:17.966616 21901 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:17.966676 21901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "75a436c503914738b9c08c867641edde"
format_stamp: "Formatted at 2025-06-26 20:07:16 on dist-test-slave-l47t"
I20250626 20:07:17.966871 21901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:17.973176 21901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:17.973394 21901 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:17.973490 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:17.976912 21901 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:35577
I20250626 20:07:17.976943 21967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:35577 every 8 connection(s)
I20250626 20:07:17.977178 21901 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:17.977990 21968 sys_catalog.cc:263] Verifying existing consensus state
I20250626 20:07:17.978726 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Bootstrap starting.
I20250626 20:07:17.982770 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 21901
I20250626 20:07:17.982860 21243 kudu-admin-test.cc:735] Forcing unsafe config change on tserver a1b9045c2112415fa364df635e08f0c8
I20250626 20:07:17.985438 21968 log.cc:826] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:17.986855 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:17.987071 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Bootstrap complete.
I20250626 20:07:17.988632 21968 raft_consensus.cc:357] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:17.988780 21968 raft_consensus.cc:738] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a436c503914738b9c08c867641edde, State: Initialized, Role: FOLLOWER
I20250626 20:07:17.988888 21968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:17.988951 21968 raft_consensus.cc:397] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:17.988991 21968 raft_consensus.cc:491] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:17.989029 21968 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:17.989648 21968 raft_consensus.cc:513] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:17.989730 21968 leader_election.cc:304] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a436c503914738b9c08c867641edde; no voters:
I20250626 20:07:17.989846 21968 leader_election.cc:290] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 20:07:17.989899 21972 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:17.990025 21972 raft_consensus.cc:695] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [term 2 LEADER]: Becoming Leader. State: Replica: 75a436c503914738b9c08c867641edde, State: Running, Role: LEADER
I20250626 20:07:17.990036 21968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:17.990129 21972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } }
I20250626 20:07:17.990558 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75a436c503914738b9c08c867641edde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } } }
I20250626 20:07:17.990648 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:17.990556 21972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a436c503914738b9c08c867641edde. Latest consensus state: current_term: 2 leader_uuid: "75a436c503914738b9c08c867641edde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a436c503914738b9c08c867641edde" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 35577 } } }
I20250626 20:07:17.990738 21972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:17.991197 21979 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:17.991711 21979 catalog_manager.cc:671] Loaded metadata for table TestTable [id=f34202be713b4f4d8c6c59f81b2adc1e]
I20250626 20:07:17.991883 21979 tablet_loader.cc:96] loaded metadata for tablet e9fd9eaebc174277ae2d8e763fe832b6 (table TestTable [id=f34202be713b4f4d8c6c59f81b2adc1e])
I20250626 20:07:17.991935 21979 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:17.992000 21979 catalog_manager.cc:1261] Loaded cluster ID: 8e840a7722d44d22b6276feb12ff9376
I20250626 20:07:17.992017 21979 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:17.993119 21979 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:17.993260 21979 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 75a436c503914738b9c08c867641edde: Loaded TSK: 0
I20250626 20:07:17.993431 21979 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 20:07:18.052100 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:18.052250 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:18.053503 21970 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 20:07:18.066237 21691 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "a1b9045c2112415fa364df635e08f0c8"
tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
}
peers {
permanent_uuid: "a1b9045c2112415fa364df635e08f0c8"
}
}
from {username='slave'} at 127.0.0.1:38060
W20250626 20:07:18.066439 21691 raft_consensus.cc:2216] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true
I20250626 20:07:18.066555 21691 raft_consensus.cc:3058] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:18.071293 21932 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" instance_seqno: 1750968437337327) as {username='slave'} at 127.20.190.195:34249; Asking this server to re-register.
I20250626 20:07:18.071668 21737 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:18.071739 21737 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:18.072032 21932 ts_manager.cc:194] Registered new tserver with Master: a1b9045c2112415fa364df635e08f0c8 (127.20.190.195:36103)
I20250626 20:07:18.133934 21475 heartbeater.cc:499] Master 127.20.190.254:35577 was elected leader, sending a full tablet report...
I20250626 20:07:18.136271 21932 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" instance_seqno: 1750968437127035) as {username='slave'} at 127.20.190.193:59947; Asking this server to re-register.
I20250626 20:07:18.136538 21475 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:18.136597 21475 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:18.136799 21932 ts_manager.cc:194] Registered new tserver with Master: be51f2f7007e443c8772aeec3964caa0 (127.20.190.193:46823)
I20250626 20:07:18.877022 21932 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" instance_seqno: 1750968437435943) as {username='slave'} at 127.20.190.196:45415; Asking this server to re-register.
I20250626 20:07:18.877225 21868 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:18.877287 21868 heartbeater.cc:507] Master 127.20.190.254:35577 requested a full tablet report, sending...
I20250626 20:07:18.877488 21932 ts_manager.cc:194] Registered new tserver with Master: 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196:41229)
I20250626 20:07:19.374656 22012 raft_consensus.cc:491] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b1dd20e1a5d4542b74bc1e25b033ce7)
I20250626 20:07:19.374727 22012 raft_consensus.cc:513] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } }
I20250626 20:07:19.374943 22012 leader_election.cc:290] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194:39351), a1b9045c2112415fa364df635e08f0c8 (127.20.190.195:36103)
W20250626 20:07:19.375252 21756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.190.194:39351: connect: Connection refused (error 111)
W20250626 20:07:19.375681 21756 leader_election.cc:336] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194:39351): Network error: Client connection negotiation failed: client connection to 127.20.190.194:39351: connect: Connection refused (error 111)
I20250626 20:07:19.377346 21691 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a1b9045c2112415fa364df635e08f0c8" is_pre_election: true
I20250626 20:07:19.377545 21756 leader_election.cc:304] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0854b1ce3f074bf580eefc095bd3e6cc; no voters: 2b1dd20e1a5d4542b74bc1e25b033ce7, a1b9045c2112415fa364df635e08f0c8
I20250626 20:07:19.377651 22012 raft_consensus.cc:2747] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 20:07:19.567910 22016 raft_consensus.cc:491] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250626 20:07:19.567982 22016 raft_consensus.cc:513] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true
I20250626 20:07:19.568224 22016 leader_election.cc:290] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196:41229)
I20250626 20:07:19.570374 21822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "a1b9045c2112415fa364df635e08f0c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" is_pre_election: true
I20250626 20:07:19.570444 21822 raft_consensus.cc:2466] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b9045c2112415fa364df635e08f0c8 in term 1.
I20250626 20:07:19.570611 21624 leader_election.cc:304] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0854b1ce3f074bf580eefc095bd3e6cc, a1b9045c2112415fa364df635e08f0c8; no voters:
I20250626 20:07:19.570725 22016 raft_consensus.cc:2802] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 20:07:19.570773 22016 raft_consensus.cc:491] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250626 20:07:19.570792 22016 raft_consensus.cc:3058] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:19.571332 22016 raft_consensus.cc:513] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true
I20250626 20:07:19.571414 22016 leader_election.cc:290] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [CANDIDATE]: Term 3 election: Requested vote from peers 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196:41229)
I20250626 20:07:19.571570 21822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6" candidate_uuid: "a1b9045c2112415fa364df635e08f0c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
I20250626 20:07:19.571624 21822 raft_consensus.cc:3058] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 1 FOLLOWER]: Advancing to term 3
I20250626 20:07:19.572064 21822 raft_consensus.cc:2466] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b9045c2112415fa364df635e08f0c8 in term 3.
I20250626 20:07:19.572206 21624 leader_election.cc:304] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0854b1ce3f074bf580eefc095bd3e6cc, a1b9045c2112415fa364df635e08f0c8; no voters:
I20250626 20:07:19.572278 22016 raft_consensus.cc:2802] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 20:07:19.572362 22016 raft_consensus.cc:695] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 3 LEADER]: Becoming Leader. State: Replica: a1b9045c2112415fa364df635e08f0c8, State: Running, Role: LEADER
I20250626 20:07:19.572427 22016 consensus_queue.cc:237] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true
I20250626 20:07:19.572994 21932 catalog_manager.cc:5582] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 reported cstate change: term changed from 1 to 3, leader changed from 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194) to a1b9045c2112415fa364df635e08f0c8 (127.20.190.195), now has a pending config: VOTER 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196), VOTER a1b9045c2112415fa364df635e08f0c8 (127.20.190.195). New cstate: current_term: 3 leader_uuid: "a1b9045c2112415fa364df635e08f0c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1dd20e1a5d4542b74bc1e25b033ce7" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 39351 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true }
I20250626 20:07:20.051959 21822 raft_consensus.cc:1273] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Refusing update from remote peer a1b9045c2112415fa364df635e08f0c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 20:07:20.052179 22016 consensus_queue.cc:1035] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 20:07:20.053056 22019 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true }
I20250626 20:07:20.053177 21822 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } unsafe_config_change: true }
I20250626 20:07:20.054050 21932 catalog_manager.cc:5582] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 reported cstate change: config changed from index -1 to 2, VOTER 2b1dd20e1a5d4542b74bc1e25b033ce7 (127.20.190.194) evicted, no longer has a pending config: VOTER 0854b1ce3f074bf580eefc095bd3e6cc (127.20.190.196), VOTER a1b9045c2112415fa364df635e08f0c8 (127.20.190.195). New cstate: current_term: 3 leader_uuid: "a1b9045c2112415fa364df635e08f0c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250626 20:07:20.054740 21932 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet e9fd9eaebc174277ae2d8e763fe832b6 on TS 2b1dd20e1a5d4542b74bc1e25b033ce7: Not found: failed to reset TS proxy: Could not find TS for UUID 2b1dd20e1a5d4542b74bc1e25b033ce7
I20250626 20:07:20.057541 21691 consensus_queue.cc:237] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: true } } unsafe_config_change: true
I20250626 20:07:20.057911 21822 raft_consensus.cc:1273] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Refusing update from remote peer a1b9045c2112415fa364df635e08f0c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250626 20:07:20.058074 22019 consensus_queue.cc:1035] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 20:07:20.058599 22016 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: true } } unsafe_config_change: true }
I20250626 20:07:20.058720 21822 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: true } } unsafe_config_change: true }
I20250626 20:07:20.059544 21932 catalog_manager.cc:5582] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc reported cstate change: config changed from index 2 to 4, NON_VOTER be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) added. New cstate: current_term: 3 leader_uuid: "a1b9045c2112415fa364df635e08f0c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: true } } unsafe_config_change: true }
I20250626 20:07:20.059924 21917 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e9fd9eaebc174277ae2d8e763fe832b6 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250626 20:07:20.060793 21626 consensus_peers.cc:487] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 -> Peer be51f2f7007e443c8772aeec3964caa0 (127.20.190.193:46823): Couldn't send request to peer be51f2f7007e443c8772aeec3964caa0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9fd9eaebc174277ae2d8e763fe832b6. This is attempt 1: this message will repeat every 5th retry.
W20250626 20:07:20.107273 21917 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet e9fd9eaebc174277ae2d8e763fe832b6 on TS 2b1dd20e1a5d4542b74bc1e25b033ce7 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2b1dd20e1a5d4542b74bc1e25b033ce7
I20250626 20:07:20.463557 22033 ts_tablet_manager.cc:927] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Initiating tablet copy from peer a1b9045c2112415fa364df635e08f0c8 (127.20.190.195:36103)
I20250626 20:07:20.463734 22033 tablet_copy_client.cc:323] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.190.195:36103
I20250626 20:07:20.466006 21711 tablet_copy_service.cc:140] P a1b9045c2112415fa364df635e08f0c8: Received BeginTabletCopySession request for tablet e9fd9eaebc174277ae2d8e763fe832b6 from peer be51f2f7007e443c8772aeec3964caa0 ({username='slave'} at 127.20.190.193:39205)
I20250626 20:07:20.466065 21711 tablet_copy_service.cc:161] P a1b9045c2112415fa364df635e08f0c8: Beginning new tablet copy session on tablet e9fd9eaebc174277ae2d8e763fe832b6 from peer be51f2f7007e443c8772aeec3964caa0 at {username='slave'} at 127.20.190.193:39205: session id = be51f2f7007e443c8772aeec3964caa0-e9fd9eaebc174277ae2d8e763fe832b6
I20250626 20:07:20.466478 21711 tablet_copy_source_session.cc:215] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 20:07:20.467165 22033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fd9eaebc174277ae2d8e763fe832b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:20.468761 22033 tablet_copy_client.cc:806] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: tablet copy: Starting download of 0 data blocks...
I20250626 20:07:20.468891 22033 tablet_copy_client.cc:670] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: tablet copy: Starting download of 1 WAL segments...
I20250626 20:07:20.469455 22033 tablet_copy_client.cc:538] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 20:07:20.469998 22033 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Bootstrap starting.
I20250626 20:07:20.471009 22033 log.cc:826] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:20.471731 22033 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:20.471865 22033 tablet_bootstrap.cc:492] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Bootstrap complete.
I20250626 20:07:20.471933 22033 ts_tablet_manager.cc:1397] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:20.472973 22033 raft_consensus.cc:357] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: true } } unsafe_config_change: true
I20250626 20:07:20.473076 22033 raft_consensus.cc:738] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: be51f2f7007e443c8772aeec3964caa0, State: Initialized, Role: LEARNER
I20250626 20:07:20.473119 22033 consensus_queue.cc:260] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: true } } unsafe_config_change: true
I20250626 20:07:20.473270 22033 ts_tablet_manager.cc:1428] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:20.473513 21711 tablet_copy_service.cc:342] P a1b9045c2112415fa364df635e08f0c8: Request end of tablet copy session be51f2f7007e443c8772aeec3964caa0-e9fd9eaebc174277ae2d8e763fe832b6 received from {username='slave'} at 127.20.190.193:39205
I20250626 20:07:20.473562 21711 tablet_copy_service.cc:434] P a1b9045c2112415fa364df635e08f0c8: ending tablet copy session be51f2f7007e443c8772aeec3964caa0-e9fd9eaebc174277ae2d8e763fe832b6 on tablet e9fd9eaebc174277ae2d8e763fe832b6 with peer be51f2f7007e443c8772aeec3964caa0
I20250626 20:07:20.906102 21429 raft_consensus.cc:1215] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
W20250626 20:07:21.272043 21917 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet e9fd9eaebc174277ae2d8e763fe832b6 on TS 2b1dd20e1a5d4542b74bc1e25b033ce7 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2b1dd20e1a5d4542b74bc1e25b033ce7
I20250626 20:07:21.363206 22019 raft_consensus.cc:1062] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8: attempting to promote NON_VOTER be51f2f7007e443c8772aeec3964caa0 to VOTER
I20250626 20:07:21.363394 22019 consensus_queue.cc:237] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: false } } unsafe_config_change: true
I20250626 20:07:21.363878 21822 raft_consensus.cc:1273] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Refusing update from remote peer a1b9045c2112415fa364df635e08f0c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 20:07:21.363929 21428 raft_consensus.cc:1273] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 [term 3 LEARNER]: Refusing update from remote peer a1b9045c2112415fa364df635e08f0c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 20:07:21.364100 22038 consensus_queue.cc:1035] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 20:07:21.364212 22019 consensus_queue.cc:1035] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 20:07:21.365044 21822 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P 0854b1ce3f074bf580eefc095bd3e6cc [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: false } } unsafe_config_change: true }
I20250626 20:07:21.365362 21428 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: false } } unsafe_config_change: true }
I20250626 20:07:21.364914 22038 raft_consensus.cc:2953] T e9fd9eaebc174277ae2d8e763fe832b6 P a1b9045c2112415fa364df635e08f0c8 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: false } } unsafe_config_change: true }
I20250626 20:07:21.366501 21931 catalog_manager.cc:5582] T e9fd9eaebc174277ae2d8e763fe832b6 P be51f2f7007e443c8772aeec3964caa0 reported cstate change: config changed from index 4 to 5, be51f2f7007e443c8772aeec3964caa0 (127.20.190.193) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a1b9045c2112415fa364df635e08f0c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc" member_type: VOTER last_known_addr { host: "127.20.190.196" port: 41229 } } peers { permanent_uuid: "a1b9045c2112415fa364df635e08f0c8" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 36103 } } peers { permanent_uuid: "be51f2f7007e443c8772aeec3964caa0" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 46823 } attrs { promote: false } } unsafe_config_change: true }
I20250626 20:07:21.420732 21243 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250626 20:07:21.427910 21243 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "e9fd9eaebc174277ae2d8e763fe832b6"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "0854b1ce3f074bf580eefc095bd3e6cc"
rpc_addresses {
host: "127.20.190.196"
port: 41229
}
}
ts_infos {
permanent_uuid: "a1b9045c2112415fa364df635e08f0c8"
rpc_addresses {
host: "127.20.190.195"
port: 36103
}
}
ts_infos {
permanent_uuid: "be51f2f7007e443c8772aeec3964caa0"
rpc_addresses {
host: "127.20.190.193"
port: 46823
}
}
I20250626 20:07:21.428195 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 21347
I20250626 20:07:21.432523 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 21609
I20250626 20:07:21.436472 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 21740
I20250626 20:07:21.440580 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 21901
2025-06-26T20:07:21Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (4579 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250626 20:07:21.461124 21243 test_util.cc:276] Using random seed: 469195138
I20250626 20:07:21.461519 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:21.461549 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:21.461560 21243 ts_itest-base.cc:117] 3 tablet servers
I20250626 20:07:21.461570 21243 ts_itest-base.cc:118] 3 replicas per TS
I20250626 20:07:21.461589 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:21Z Disabled control of system clock
I20250626 20:07:21.467479 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:43077
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:43077
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250626 20:07:21.534381 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:21.534534 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:21.534562 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:21.535780 22058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:21.535820 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:21.535848 22058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:21.535868 22058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:21.537237 22058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:43077
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:43077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:21.537420 22058 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:21.537592 22058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:21.540409 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.540467 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.540474 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:21.540616 22058 server_base.cc:1048] running on GCE node
I20250626 20:07:21.540802 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:21.540979 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:21.542093 22058 hybrid_clock.cc:648] HybridClock initialized: now 1750968441542080 us; error 24 us; skew 500 ppm
I20250626 20:07:21.543087 22058 webserver.cc:469] Webserver started at http://127.20.190.254:41329/ using document root <none> and password file <none>
I20250626 20:07:21.543244 22058 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:21.543282 22058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:21.543370 22058 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:21.544005 22058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "949ba51d7170471ea1918885deb9e17d"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.544239 22058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "949ba51d7170471ea1918885deb9e17d"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.545183 22058 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:21.545729 22072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.545876 22058 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:21.545943 22058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "949ba51d7170471ea1918885deb9e17d"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.545996 22058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:21.560328 22058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:21.560549 22058 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:21.560645 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:21.563730 22058 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:43077
I20250626 20:07:21.563758 22124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:43077 every 8 connection(s)
I20250626 20:07:21.563980 22058 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:21.564409 22125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:21.566362 22125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d: Bootstrap starting.
I20250626 20:07:21.566918 22125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:21.567133 22125 log.cc:826] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:21.567622 22125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d: No bootstrap required, opened a new log
I20250626 20:07:21.568580 22125 raft_consensus.cc:357] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ba51d7170471ea1918885deb9e17d" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 43077 } }
I20250626 20:07:21.568677 22125 raft_consensus.cc:383] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:21.568707 22125 raft_consensus.cc:738] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949ba51d7170471ea1918885deb9e17d, State: Initialized, Role: FOLLOWER
I20250626 20:07:21.568820 22125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ba51d7170471ea1918885deb9e17d" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 43077 } }
I20250626 20:07:21.568886 22125 raft_consensus.cc:397] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:21.568928 22125 raft_consensus.cc:491] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:21.568966 22125 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:21.569408 22125 raft_consensus.cc:513] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ba51d7170471ea1918885deb9e17d" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 43077 } }
I20250626 20:07:21.569490 22125 leader_election.cc:304] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 949ba51d7170471ea1918885deb9e17d; no voters:
I20250626 20:07:21.569610 22125 leader_election.cc:290] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:21.569643 22128 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:21.569730 22128 raft_consensus.cc:695] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [term 1 LEADER]: Becoming Leader. State: Replica: 949ba51d7170471ea1918885deb9e17d, State: Running, Role: LEADER
I20250626 20:07:21.569790 22125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:21.569836 22128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ba51d7170471ea1918885deb9e17d" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 43077 } }
I20250626 20:07:21.570282 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22058
I20250626 20:07:21.570343 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:21.570598 22129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "949ba51d7170471ea1918885deb9e17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ba51d7170471ea1918885deb9e17d" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 43077 } } }
I20250626 20:07:21.570580 22130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949ba51d7170471ea1918885deb9e17d. Latest consensus state: current_term: 1 leader_uuid: "949ba51d7170471ea1918885deb9e17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ba51d7170471ea1918885deb9e17d" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 43077 } } }
I20250626 20:07:21.570683 22129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:21.570686 22130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:21.571240 22135 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:21.571631 22135 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:21.573462 22135 catalog_manager.cc:1349] Generated new cluster ID: 3bbcec16d33149348ac4eb4c3e40acf8
I20250626 20:07:21.573509 22135 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:21.586858 22135 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:21.587244 22135 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:21.597842 22135 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 949ba51d7170471ea1918885deb9e17d: Generated new TSK 0
I20250626 20:07:21.597995 22135 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:21.607887 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:43077
--builtin_ntp_servers=127.20.190.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250626 20:07:21.672533 22149 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 20:07:21.672665 22149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:21.672686 22149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:21.672711 22149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:21.673839 22149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:21.673887 22149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:21.675098 22149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:43077
--never_fsync=true
--heap_profile_path=/tmp/kudu.22149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:21.675290 22149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:21.675457 22149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:21.678994 22157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.679056 22154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.679065 22155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:21.679226 22149 server_base.cc:1048] running on GCE node
I20250626 20:07:21.679347 22149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:21.679497 22149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:21.680619 22149 hybrid_clock.cc:648] HybridClock initialized: now 1750968441680606 us; error 25 us; skew 500 ppm
I20250626 20:07:21.681587 22149 webserver.cc:469] Webserver started at http://127.20.190.193:40427/ using document root <none> and password file <none>
I20250626 20:07:21.681748 22149 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:21.681782 22149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:21.681859 22149 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:21.682487 22149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1c25a3fe319d4c9a994657f3dcee17a1"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.682766 22149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1c25a3fe319d4c9a994657f3dcee17a1"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.683676 22149 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:21.684208 22163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.684352 22149 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:21.684415 22149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1c25a3fe319d4c9a994657f3dcee17a1"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.684471 22149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:21.692499 22149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:21.692713 22149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:21.692802 22149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:21.692965 22149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:21.693187 22149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:21.693213 22149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.693239 22149 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:21.693264 22149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.698283 22149 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:44435
I20250626 20:07:21.698333 22276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:44435 every 8 connection(s)
I20250626 20:07:21.698550 22149 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:21.701295 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22149
I20250626 20:07:21.701370 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:21.702136 22277 heartbeater.cc:344] Connected to a master server at 127.20.190.254:43077
I20250626 20:07:21.702215 22277 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:21.702237 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:43077
--builtin_ntp_servers=127.20.190.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 20:07:21.702407 22277 heartbeater.cc:507] Master 127.20.190.254:43077 requested a full tablet report, sending...
I20250626 20:07:21.702804 22089 ts_manager.cc:194] Registered new tserver with Master: 1c25a3fe319d4c9a994657f3dcee17a1 (127.20.190.193:44435)
I20250626 20:07:21.703531 22089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:50871
W20250626 20:07:21.766705 22280 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 20:07:21.766824 22280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:21.766846 22280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:21.766868 22280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:21.767983 22280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:21.768030 22280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:21.769186 22280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:43077
--never_fsync=true
--heap_profile_path=/tmp/kudu.22280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:21.769371 22280 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:21.769605 22280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:21.772689 22288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.772689 22286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.772812 22285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:21.772858 22280 server_base.cc:1048] running on GCE node
I20250626 20:07:21.772976 22280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:21.773105 22280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:21.774209 22280 hybrid_clock.cc:648] HybridClock initialized: now 1750968441774203 us; error 23 us; skew 500 ppm
I20250626 20:07:21.775184 22280 webserver.cc:469] Webserver started at http://127.20.190.194:39911/ using document root <none> and password file <none>
I20250626 20:07:21.775348 22280 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:21.775385 22280 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:21.775468 22280 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:21.776193 22280 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b9508826e8b34294aab8f5317f0001e0"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.776461 22280 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b9508826e8b34294aab8f5317f0001e0"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.777415 22280 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:21.777915 22294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.778038 22280 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:21.778095 22280 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b9508826e8b34294aab8f5317f0001e0"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.778139 22280 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:21.785434 22280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:21.785681 22280 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:21.785781 22280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:21.785955 22280 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:21.786188 22280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:21.786218 22280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.786245 22280 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:21.786273 22280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.790961 22280 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:46173
I20250626 20:07:21.791018 22407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:46173 every 8 connection(s)
I20250626 20:07:21.791209 22280 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:21.794440 22408 heartbeater.cc:344] Connected to a master server at 127.20.190.254:43077
I20250626 20:07:21.794498 22408 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:21.794651 22408 heartbeater.cc:507] Master 127.20.190.254:43077 requested a full tablet report, sending...
I20250626 20:07:21.794912 22089 ts_manager.cc:194] Registered new tserver with Master: b9508826e8b34294aab8f5317f0001e0 (127.20.190.194:46173)
I20250626 20:07:21.795217 22089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:49639
I20250626 20:07:21.795653 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22280
I20250626 20:07:21.795728 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 20:07:21.796574 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:43077
--builtin_ntp_servers=127.20.190.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250626 20:07:21.859561 22411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 20:07:21.859690 22411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:21.859709 22411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:21.859735 22411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:21.860839 22411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:21.860885 22411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:21.862069 22411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:43077
--never_fsync=true
--heap_profile_path=/tmp/kudu.22411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:21.862275 22411 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:21.862444 22411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:21.865959 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:21.865962 22416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:21.866117 22411 server_base.cc:1048] running on GCE node
W20250626 20:07:21.866223 22417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:21.866366 22411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:21.866546 22411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:21.867666 22411 hybrid_clock.cc:648] HybridClock initialized: now 1750968441867660 us; error 23 us; skew 500 ppm
I20250626 20:07:21.868598 22411 webserver.cc:469] Webserver started at http://127.20.190.195:35825/ using document root <none> and password file <none>
I20250626 20:07:21.868755 22411 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:21.868790 22411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:21.868876 22411 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:21.869475 22411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b8d84597c49f4255a803269c93885c94"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.869712 22411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b8d84597c49f4255a803269c93885c94"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.870630 22411 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:21.871155 22425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.871273 22411 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:21.871327 22411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b8d84597c49f4255a803269c93885c94"
format_stamp: "Formatted at 2025-06-26 20:07:21 on dist-test-slave-l47t"
I20250626 20:07:21.871374 22411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:21.881310 22411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:21.881500 22411 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:21.881587 22411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:21.881742 22411 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:21.881951 22411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:21.881978 22411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.882002 22411 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:21.882025 22411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:21.886936 22411 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:45625
I20250626 20:07:21.886981 22538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:45625 every 8 connection(s)
I20250626 20:07:21.887172 22411 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:21.889575 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22411
I20250626 20:07:21.889650 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 20:07:21.890753 22539 heartbeater.cc:344] Connected to a master server at 127.20.190.254:43077
I20250626 20:07:21.890826 22539 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:21.890941 22539 heartbeater.cc:507] Master 127.20.190.254:43077 requested a full tablet report, sending...
I20250626 20:07:21.891189 22089 ts_manager.cc:194] Registered new tserver with Master: b8d84597c49f4255a803269c93885c94 (127.20.190.195:45625)
I20250626 20:07:21.891471 22089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:58983
I20250626 20:07:21.901127 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:21.905364 22089 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36538:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 20:07:21.906463 22089 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 20:07:21.911597 22473 tablet_service.cc:1468] Processing CreateTablet for tablet 23d9e7797d834e02939473442628c3b3 (DEFAULT_TABLE table=TestTable [id=e50280fcd27745dfadf864b50417baf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:21.911834 22473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d9e7797d834e02939473442628c3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:21.912000 22342 tablet_service.cc:1468] Processing CreateTablet for tablet 23d9e7797d834e02939473442628c3b3 (DEFAULT_TABLE table=TestTable [id=e50280fcd27745dfadf864b50417baf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:21.912088 22211 tablet_service.cc:1468] Processing CreateTablet for tablet 23d9e7797d834e02939473442628c3b3 (DEFAULT_TABLE table=TestTable [id=e50280fcd27745dfadf864b50417baf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:21.912223 22342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d9e7797d834e02939473442628c3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:21.912273 22211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d9e7797d834e02939473442628c3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:21.913342 22553 tablet_bootstrap.cc:492] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: Bootstrap starting.
I20250626 20:07:21.913851 22553 tablet_bootstrap.cc:654] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:21.914031 22553 log.cc:826] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:21.914469 22553 tablet_bootstrap.cc:492] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: No bootstrap required, opened a new log
I20250626 20:07:21.914543 22553 ts_tablet_manager.cc:1397] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:21.914644 22556 tablet_bootstrap.cc:492] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: Bootstrap starting.
I20250626 20:07:21.915163 22556 tablet_bootstrap.cc:654] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:21.915424 22556 log.cc:826] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:21.915772 22555 tablet_bootstrap.cc:492] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: Bootstrap starting.
I20250626 20:07:21.915964 22553 raft_consensus.cc:357] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.916085 22556 tablet_bootstrap.cc:492] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: No bootstrap required, opened a new log
I20250626 20:07:21.916112 22553 raft_consensus.cc:738] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d84597c49f4255a803269c93885c94, State: Initialized, Role: FOLLOWER
I20250626 20:07:21.916134 22556 ts_tablet_manager.cc:1397] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:21.916177 22553 consensus_queue.cc:260] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.916376 22555 tablet_bootstrap.cc:654] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:21.916397 22553 ts_tablet_manager.cc:1428] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:21.916401 22539 heartbeater.cc:499] Master 127.20.190.254:43077 was elected leader, sending a full tablet report...
I20250626 20:07:21.916687 22555 log.cc:826] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:21.917203 22555 tablet_bootstrap.cc:492] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: No bootstrap required, opened a new log
I20250626 20:07:21.917263 22555 ts_tablet_manager.cc:1397] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:21.917506 22556 raft_consensus.cc:357] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.917618 22556 raft_consensus.cc:738] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9508826e8b34294aab8f5317f0001e0, State: Initialized, Role: FOLLOWER
I20250626 20:07:21.917672 22556 consensus_queue.cc:260] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.917866 22556 ts_tablet_manager.cc:1428] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:21.917884 22408 heartbeater.cc:499] Master 127.20.190.254:43077 was elected leader, sending a full tablet report...
I20250626 20:07:21.918447 22555 raft_consensus.cc:357] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.918592 22555 raft_consensus.cc:738] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c25a3fe319d4c9a994657f3dcee17a1, State: Initialized, Role: FOLLOWER
I20250626 20:07:21.918648 22555 consensus_queue.cc:260] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.918828 22555 ts_tablet_manager.cc:1428] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:21.918866 22277 heartbeater.cc:499] Master 127.20.190.254:43077 was elected leader, sending a full tablet report...
I20250626 20:07:21.919622 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:21.919943 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1c25a3fe319d4c9a994657f3dcee17a1 to finish bootstrapping
I20250626 20:07:21.922369 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b9508826e8b34294aab8f5317f0001e0 to finish bootstrapping
I20250626 20:07:21.924521 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b8d84597c49f4255a803269c93885c94 to finish bootstrapping
I20250626 20:07:21.933401 22493 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "23d9e7797d834e02939473442628c3b3"
dest_uuid: "b8d84597c49f4255a803269c93885c94"
from {username='slave'} at 127.0.0.1:48292
I20250626 20:07:21.933462 22493 raft_consensus.cc:491] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 20:07:21.933487 22493 raft_consensus.cc:3058] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:21.934110 22493 raft_consensus.cc:513] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.934280 22493 leader_election.cc:290] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [CANDIDATE]: Term 1 election: Requested vote from peers b9508826e8b34294aab8f5317f0001e0 (127.20.190.194:46173), 1c25a3fe319d4c9a994657f3dcee17a1 (127.20.190.193:44435)
I20250626 20:07:21.935323 21243 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250626 20:07:21.936822 22362 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23d9e7797d834e02939473442628c3b3" candidate_uuid: "b8d84597c49f4255a803269c93885c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9508826e8b34294aab8f5317f0001e0"
I20250626 20:07:21.936887 22362 raft_consensus.cc:3058] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:21.937198 22231 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23d9e7797d834e02939473442628c3b3" candidate_uuid: "b8d84597c49f4255a803269c93885c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1c25a3fe319d4c9a994657f3dcee17a1"
I20250626 20:07:21.937258 22231 raft_consensus.cc:3058] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:21.937610 22362 raft_consensus.cc:2466] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d84597c49f4255a803269c93885c94 in term 1.
I20250626 20:07:21.937706 22231 raft_consensus.cc:2466] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d84597c49f4255a803269c93885c94 in term 1.
I20250626 20:07:21.937793 22427 leader_election.cc:304] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d84597c49f4255a803269c93885c94, b9508826e8b34294aab8f5317f0001e0; no voters:
I20250626 20:07:21.937894 22558 raft_consensus.cc:2802] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:21.937997 22558 raft_consensus.cc:695] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 LEADER]: Becoming Leader. State: Replica: b8d84597c49f4255a803269c93885c94, State: Running, Role: LEADER
I20250626 20:07:21.938056 22558 consensus_queue.cc:237] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:21.938565 22089 catalog_manager.cc:5582] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8d84597c49f4255a803269c93885c94 (127.20.190.195). New cstate: current_term: 1 leader_uuid: "b8d84597c49f4255a803269c93885c94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } health_report { overall_health: UNKNOWN } } }
W20250626 20:07:21.949026 22278 tablet.cc:2378] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:22.036501 21243 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250626 20:07:22.041673 22409 tablet.cc:2378] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:22.137653 22540 tablet.cc:2378] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:22.237736 21243 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 20:07:22.342705 22558 consensus_queue.cc:1035] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:22.346282 22564 consensus_queue.cc:1035] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:22.618471 22493 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "23d9e7797d834e02939473442628c3b3"
dest_uuid: "b8d84597c49f4255a803269c93885c94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48308
I20250626 20:07:22.618562 22493 raft_consensus.cc:604] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 LEADER]: Received request to transfer leadership
I20250626 20:07:22.843464 22564 raft_consensus.cc:991] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94: : Instructing follower b9508826e8b34294aab8f5317f0001e0 to start an election
I20250626 20:07:22.843538 22564 raft_consensus.cc:1079] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 LEADER]: Signalling peer b9508826e8b34294aab8f5317f0001e0 to start an election
I20250626 20:07:22.843780 22362 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "23d9e7797d834e02939473442628c3b3"
dest_uuid: "b9508826e8b34294aab8f5317f0001e0"
from {username='slave'} at 127.20.190.195:49891
I20250626 20:07:22.843860 22362 raft_consensus.cc:491] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 20:07:22.843880 22362 raft_consensus.cc:3058] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:22.844424 22362 raft_consensus.cc:513] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:22.844650 22362 leader_election.cc:290] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [CANDIDATE]: Term 2 election: Requested vote from peers b8d84597c49f4255a803269c93885c94 (127.20.190.195:45625), 1c25a3fe319d4c9a994657f3dcee17a1 (127.20.190.193:44435)
I20250626 20:07:22.847281 22493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23d9e7797d834e02939473442628c3b3" candidate_uuid: "b9508826e8b34294aab8f5317f0001e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b8d84597c49f4255a803269c93885c94"
I20250626 20:07:22.847355 22493 raft_consensus.cc:3053] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 20:07:22.847378 22493 raft_consensus.cc:738] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8d84597c49f4255a803269c93885c94, State: Running, Role: LEADER
I20250626 20:07:22.847414 22493 consensus_queue.cc:260] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:22.847398 22231 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23d9e7797d834e02939473442628c3b3" candidate_uuid: "b9508826e8b34294aab8f5317f0001e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c25a3fe319d4c9a994657f3dcee17a1"
I20250626 20:07:22.847463 22231 raft_consensus.cc:3058] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:22.847503 22493 raft_consensus.cc:3058] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:22.848078 22231 raft_consensus.cc:2466] T 23d9e7797d834e02939473442628c3b3 P 1c25a3fe319d4c9a994657f3dcee17a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9508826e8b34294aab8f5317f0001e0 in term 2.
I20250626 20:07:22.848078 22493 raft_consensus.cc:2466] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9508826e8b34294aab8f5317f0001e0 in term 2.
I20250626 20:07:22.848241 22295 leader_election.cc:304] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d84597c49f4255a803269c93885c94, b9508826e8b34294aab8f5317f0001e0; no voters:
I20250626 20:07:22.848399 22592 raft_consensus.cc:2802] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:22.848492 22592 raft_consensus.cc:695] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [term 2 LEADER]: Becoming Leader. State: Replica: b9508826e8b34294aab8f5317f0001e0, State: Running, Role: LEADER
I20250626 20:07:22.848570 22592 consensus_queue.cc:237] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } }
I20250626 20:07:22.849098 22089 catalog_manager.cc:5582] T 23d9e7797d834e02939473442628c3b3 P b9508826e8b34294aab8f5317f0001e0 reported cstate change: term changed from 1 to 2, leader changed from b8d84597c49f4255a803269c93885c94 (127.20.190.195) to b9508826e8b34294aab8f5317f0001e0 (127.20.190.194). New cstate: current_term: 2 leader_uuid: "b9508826e8b34294aab8f5317f0001e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9508826e8b34294aab8f5317f0001e0" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 46173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8d84597c49f4255a803269c93885c94" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 45625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c25a3fe319d4c9a994657f3dcee17a1" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 44435 } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:22.963063 22493 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "23d9e7797d834e02939473442628c3b3"
dest_uuid: "b8d84597c49f4255a803269c93885c94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48318
I20250626 20:07:22.963129 22493 raft_consensus.cc:604] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 2 FOLLOWER]: Received request to transfer leadership
I20250626 20:07:22.963146 22493 raft_consensus.cc:612] T 23d9e7797d834e02939473442628c3b3 P b8d84597c49f4255a803269c93885c94 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 20:07:22.967730 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22149
I20250626 20:07:22.971861 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22280
I20250626 20:07:22.975962 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22411
I20250626 20:07:22.979614 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22058
2025-06-26T20:07:22Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (1536 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250626 20:07:22.997856 21243 test_util.cc:276] Using random seed: 470731870
I20250626 20:07:22.998252 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:22.998283 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:22.998297 21243 ts_itest-base.cc:117] 3 tablet servers
I20250626 20:07:22.998312 21243 ts_itest-base.cc:118] 3 replicas per TS
I20250626 20:07:22.998334 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:22Z Disabled control of system clock
I20250626 20:07:23.003980 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:41661
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:41661 with env {}
W20250626 20:07:23.068868 22619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.068993 22619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:23.069020 22619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.070163 22619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:23.070201 22619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:23.070221 22619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:23.070236 22619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:23.071478 22619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:41661
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:41661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:23.071673 22619 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:23.071849 22619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:23.074827 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.074904 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.074829 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:23.074944 22619 server_base.cc:1048] running on GCE node
I20250626 20:07:23.075169 22619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:23.075357 22619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:23.076478 22619 hybrid_clock.cc:648] HybridClock initialized: now 1750968443076464 us; error 25 us; skew 500 ppm
I20250626 20:07:23.077425 22619 webserver.cc:469] Webserver started at http://127.20.190.254:34839/ using document root <none> and password file <none>
I20250626 20:07:23.077574 22619 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:23.077611 22619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:23.077701 22619 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:23.078366 22619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7a01527cab72457b970958c22ca53808"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.078641 22619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7a01527cab72457b970958c22ca53808"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.079586 22619 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 20:07:23.080098 22633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.080231 22619 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:23.080291 22619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7a01527cab72457b970958c22ca53808"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.080338 22619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:23.094658 22619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:23.094888 22619 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:23.094982 22619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:23.098289 22619 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:41661
I20250626 20:07:23.098371 22685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:41661 every 8 connection(s)
I20250626 20:07:23.098616 22619 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:23.099069 22686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.100893 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808: Bootstrap starting.
I20250626 20:07:23.101331 22686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.101503 22686 log.cc:826] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:23.101934 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808: No bootstrap required, opened a new log
I20250626 20:07:23.102866 22686 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a01527cab72457b970958c22ca53808" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41661 } }
I20250626 20:07:23.102959 22686 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.102982 22686 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a01527cab72457b970958c22ca53808, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.103049 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a01527cab72457b970958c22ca53808" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41661 } }
I20250626 20:07:23.103086 22686 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:23.103108 22686 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:23.103132 22686 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.103515 22686 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a01527cab72457b970958c22ca53808" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41661 } }
I20250626 20:07:23.103585 22686 leader_election.cc:304] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a01527cab72457b970958c22ca53808; no voters:
I20250626 20:07:23.103680 22686 leader_election.cc:290] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:23.103724 22689 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:23.103842 22686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:23.103874 22689 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [term 1 LEADER]: Becoming Leader. State: Replica: 7a01527cab72457b970958c22ca53808, State: Running, Role: LEADER
I20250626 20:07:23.103974 22689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a01527cab72457b970958c22ca53808" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41661 } }
I20250626 20:07:23.104214 22691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a01527cab72457b970958c22ca53808. Latest consensus state: current_term: 1 leader_uuid: "7a01527cab72457b970958c22ca53808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a01527cab72457b970958c22ca53808" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41661 } } }
I20250626 20:07:23.104215 22690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a01527cab72457b970958c22ca53808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a01527cab72457b970958c22ca53808" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41661 } } }
I20250626 20:07:23.104346 22691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:23.104728 22690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:23.104740 22698 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:23.105155 22698 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:23.106693 22698 catalog_manager.cc:1349] Generated new cluster ID: bb5659058fd34b9aa4c93db640227286
I20250626 20:07:23.106745 22698 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:23.107014 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22619
I20250626 20:07:23.107086 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:23.118913 22698 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:23.119304 22698 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:23.123281 22698 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7a01527cab72457b970958c22ca53808: Generated new TSK 0
I20250626 20:07:23.123391 22698 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:23.127600 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41661
--builtin_ntp_servers=127.20.190.212:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:23.199126 22710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.199240 22710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:23.199266 22710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.200470 22710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:23.200518 22710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:23.201712 22710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41661
--never_fsync=true
--heap_profile_path=/tmp/kudu.22710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:23.201890 22710 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:23.202055 22710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:23.205327 22716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:23.205461 22710 server_base.cc:1048] running on GCE node
W20250626 20:07:23.205466 22718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.205518 22715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:23.205760 22710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:23.205935 22710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:23.207051 22710 hybrid_clock.cc:648] HybridClock initialized: now 1750968443207040 us; error 29 us; skew 500 ppm
I20250626 20:07:23.208001 22710 webserver.cc:469] Webserver started at http://127.20.190.193:43251/ using document root <none> and password file <none>
I20250626 20:07:23.208148 22710 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:23.208185 22710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:23.208274 22710 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:23.208917 22710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2c3ef21adb814670a8542e9b26b95776"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.209144 22710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2c3ef21adb814670a8542e9b26b95776"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.210105 22710 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:23.210649 22724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.210805 22710 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.210863 22710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c3ef21adb814670a8542e9b26b95776"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.210911 22710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:23.219444 22710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:23.219640 22710 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:23.219739 22710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:23.219885 22710 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:23.220095 22710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:23.220121 22710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.220146 22710 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:23.220171 22710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.224918 22710 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:37073
I20250626 20:07:23.224970 22837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:37073 every 8 connection(s)
I20250626 20:07:23.225181 22710 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:23.229212 22838 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41661
I20250626 20:07:23.229283 22838 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:23.229444 22838 heartbeater.cc:507] Master 127.20.190.254:41661 requested a full tablet report, sending...
I20250626 20:07:23.229804 22650 ts_manager.cc:194] Registered new tserver with Master: 2c3ef21adb814670a8542e9b26b95776 (127.20.190.193:37073)
I20250626 20:07:23.230336 22650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:41059
I20250626 20:07:23.231241 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22710
I20250626 20:07:23.231341 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:23.232306 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41661
--builtin_ntp_servers=127.20.190.212:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:23.302851 22841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.302966 22841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:23.302982 22841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.304093 22841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:23.304133 22841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:23.305389 22841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41661
--never_fsync=true
--heap_profile_path=/tmp/kudu.22841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:23.305550 22841 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:23.305744 22841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:23.308636 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.308698 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.308779 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:23.308862 22841 server_base.cc:1048] running on GCE node
I20250626 20:07:23.308979 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:23.309111 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:23.310218 22841 hybrid_clock.cc:648] HybridClock initialized: now 1750968443310207 us; error 24 us; skew 500 ppm
I20250626 20:07:23.311162 22841 webserver.cc:469] Webserver started at http://127.20.190.194:36015/ using document root <none> and password file <none>
I20250626 20:07:23.311333 22841 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:23.311378 22841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:23.311487 22841 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:23.312140 22841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d20b583c8a7e434090db8ab6a8e540bf"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.312373 22841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d20b583c8a7e434090db8ab6a8e540bf"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.313274 22841 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:23.313809 22855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.313930 22841 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:23.313987 22841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d20b583c8a7e434090db8ab6a8e540bf"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.314033 22841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:23.319454 22841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:23.319645 22841 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:23.319747 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:23.319896 22841 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:23.320106 22841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:23.320138 22841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.320163 22841 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:23.320187 22841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.324703 22841 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:45509
I20250626 20:07:23.324759 22968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:45509 every 8 connection(s)
I20250626 20:07:23.324932 22841 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:23.325944 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22841
I20250626 20:07:23.326053 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 20:07:23.327008 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41661
--builtin_ntp_servers=127.20.190.212:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:23.328896 22969 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41661
I20250626 20:07:23.328970 22969 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:23.329108 22969 heartbeater.cc:507] Master 127.20.190.254:41661 requested a full tablet report, sending...
I20250626 20:07:23.329388 22650 ts_manager.cc:194] Registered new tserver with Master: d20b583c8a7e434090db8ab6a8e540bf (127.20.190.194:45509)
I20250626 20:07:23.329699 22650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:41443
W20250626 20:07:23.389565 22972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.389675 22972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:23.389691 22972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.390826 22972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:23.390866 22972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:23.392032 22972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41661
--never_fsync=true
--heap_profile_path=/tmp/kudu.22972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:23.392206 22972 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:23.392410 22972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:23.395196 22980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.395208 22978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:23.395268 22977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:23.395301 22972 server_base.cc:1048] running on GCE node
I20250626 20:07:23.395503 22972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:23.395677 22972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:23.396799 22972 hybrid_clock.cc:648] HybridClock initialized: now 1750968443396791 us; error 24 us; skew 500 ppm
I20250626 20:07:23.397632 22972 webserver.cc:469] Webserver started at http://127.20.190.195:42865/ using document root <none> and password file <none>
I20250626 20:07:23.397778 22972 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:23.397810 22972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:23.397895 22972 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:23.398545 22972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7ede26a2a4f04c20aff525120a238252"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.398838 22972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7ede26a2a4f04c20aff525120a238252"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.399753 22972 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 20:07:23.400247 22986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.400383 22972 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:23.400439 22972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7ede26a2a4f04c20aff525120a238252"
format_stamp: "Formatted at 2025-06-26 20:07:23 on dist-test-slave-l47t"
I20250626 20:07:23.400475 22972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:23.408458 22972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:23.408643 22972 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:23.408715 22972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:23.408845 22972 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:23.409054 22972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:23.409080 22972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.409108 22972 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:23.409132 22972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.413663 22972 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:33615
I20250626 20:07:23.413714 23099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:33615 every 8 connection(s)
I20250626 20:07:23.413911 22972 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:23.417202 23100 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41661
I20250626 20:07:23.417269 23100 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:23.417388 23100 heartbeater.cc:507] Master 127.20.190.254:41661 requested a full tablet report, sending...
I20250626 20:07:23.417630 22650 ts_manager.cc:194] Registered new tserver with Master: 7ede26a2a4f04c20aff525120a238252 (127.20.190.195:33615)
I20250626 20:07:23.417917 22650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:38785
I20250626 20:07:23.420579 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 22972
I20250626 20:07:23.420647 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 20:07:23.421291 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:23.425325 22650 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33892:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 20:07:23.426416 22650 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 20:07:23.431075 22772 tablet_service.cc:1468] Processing CreateTablet for tablet 0259266badac49bf8560179b45ea0dd2 (DEFAULT_TABLE table=TestTable [id=5715be5c16e54329a93dec9ca213604f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:23.431313 22772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0259266badac49bf8560179b45ea0dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.431947 22903 tablet_service.cc:1468] Processing CreateTablet for tablet 0259266badac49bf8560179b45ea0dd2 (DEFAULT_TABLE table=TestTable [id=5715be5c16e54329a93dec9ca213604f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:23.432111 22903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0259266badac49bf8560179b45ea0dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.432998 23114 tablet_bootstrap.cc:492] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: Bootstrap starting.
I20250626 20:07:23.433581 23114 tablet_bootstrap.cc:654] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.433892 23114 log.cc:826] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:23.434154 23115 tablet_bootstrap.cc:492] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: Bootstrap starting.
I20250626 20:07:23.434274 23034 tablet_service.cc:1468] Processing CreateTablet for tablet 0259266badac49bf8560179b45ea0dd2 (DEFAULT_TABLE table=TestTable [id=5715be5c16e54329a93dec9ca213604f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:23.434474 23034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0259266badac49bf8560179b45ea0dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.434865 23115 tablet_bootstrap.cc:654] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.435025 23114 tablet_bootstrap.cc:492] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: No bootstrap required, opened a new log
I20250626 20:07:23.435072 23114 ts_tablet_manager.cc:1397] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:23.435096 23115 log.cc:826] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:23.435674 23115 tablet_bootstrap.cc:492] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: No bootstrap required, opened a new log
I20250626 20:07:23.435724 23115 ts_tablet_manager.cc:1397] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 20:07:23.436256 23118 tablet_bootstrap.cc:492] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: Bootstrap starting.
I20250626 20:07:23.436462 23114 raft_consensus.cc:357] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.436602 23114 raft_consensus.cc:383] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.436632 23114 raft_consensus.cc:738] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3ef21adb814670a8542e9b26b95776, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.436714 23114 consensus_queue.cc:260] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.436771 23118 tablet_bootstrap.cc:654] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.436932 23114 ts_tablet_manager.cc:1428] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:23.436973 22838 heartbeater.cc:499] Master 127.20.190.254:41661 was elected leader, sending a full tablet report...
I20250626 20:07:23.436986 23118 log.cc:826] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:23.437331 23115 raft_consensus.cc:357] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.437415 23115 raft_consensus.cc:383] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.437440 23115 raft_consensus.cc:738] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20b583c8a7e434090db8ab6a8e540bf, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.437510 23118 tablet_bootstrap.cc:492] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: No bootstrap required, opened a new log
I20250626 20:07:23.437529 23115 consensus_queue.cc:260] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.437565 23118 ts_tablet_manager.cc:1397] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:23.437734 23115 ts_tablet_manager.cc:1428] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:23.437744 22969 heartbeater.cc:499] Master 127.20.190.254:41661 was elected leader, sending a full tablet report...
I20250626 20:07:23.438695 23118 raft_consensus.cc:357] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.438782 23118 raft_consensus.cc:383] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.438798 23118 raft_consensus.cc:738] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ede26a2a4f04c20aff525120a238252, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.438845 23118 consensus_queue.cc:260] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.438977 23118 ts_tablet_manager.cc:1428] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:23.439020 23100 heartbeater.cc:499] Master 127.20.190.254:41661 was elected leader, sending a full tablet report...
W20250626 20:07:23.475561 22839 tablet.cc:2378] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:23.486989 23122 raft_consensus.cc:491] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:23.487049 23122 raft_consensus.cc:513] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.487253 23122 leader_election.cc:290] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20b583c8a7e434090db8ab6a8e540bf (127.20.190.194:45509), 2c3ef21adb814670a8542e9b26b95776 (127.20.190.193:37073)
I20250626 20:07:23.489544 22923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0259266badac49bf8560179b45ea0dd2" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20b583c8a7e434090db8ab6a8e540bf" is_pre_election: true
I20250626 20:07:23.489624 22923 raft_consensus.cc:2466] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 0.
I20250626 20:07:23.489806 22990 leader_election.cc:304] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ede26a2a4f04c20aff525120a238252, d20b583c8a7e434090db8ab6a8e540bf; no voters:
I20250626 20:07:23.489799 22792 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0259266badac49bf8560179b45ea0dd2" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3ef21adb814670a8542e9b26b95776" is_pre_election: true
I20250626 20:07:23.489882 22792 raft_consensus.cc:2466] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 0.
I20250626 20:07:23.489893 23122 raft_consensus.cc:2802] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:23.489935 23122 raft_consensus.cc:491] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:23.489965 23122 raft_consensus.cc:3058] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.490422 23122 raft_consensus.cc:513] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.490553 23122 leader_election.cc:290] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 election: Requested vote from peers d20b583c8a7e434090db8ab6a8e540bf (127.20.190.194:45509), 2c3ef21adb814670a8542e9b26b95776 (127.20.190.193:37073)
I20250626 20:07:23.490629 22923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0259266badac49bf8560179b45ea0dd2" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20b583c8a7e434090db8ab6a8e540bf"
I20250626 20:07:23.490701 22923 raft_consensus.cc:3058] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.490689 22792 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0259266badac49bf8560179b45ea0dd2" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3ef21adb814670a8542e9b26b95776"
I20250626 20:07:23.490741 22792 raft_consensus.cc:3058] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.491374 22923 raft_consensus.cc:2466] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 1.
I20250626 20:07:23.491509 22990 leader_election.cc:304] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ede26a2a4f04c20aff525120a238252, d20b583c8a7e434090db8ab6a8e540bf; no voters:
I20250626 20:07:23.491597 23122 raft_consensus.cc:2802] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:23.491643 22792 raft_consensus.cc:2466] T 0259266badac49bf8560179b45ea0dd2 P 2c3ef21adb814670a8542e9b26b95776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 1.
I20250626 20:07:23.491736 23122 raft_consensus.cc:695] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [term 1 LEADER]: Becoming Leader. State: Replica: 7ede26a2a4f04c20aff525120a238252, State: Running, Role: LEADER
I20250626 20:07:23.491809 23122 consensus_queue.cc:237] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } }
I20250626 20:07:23.492283 22650 catalog_manager.cc:5582] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ede26a2a4f04c20aff525120a238252 (127.20.190.195). New cstate: current_term: 1 leader_uuid: "7ede26a2a4f04c20aff525120a238252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:23.500819 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:23.501169 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2c3ef21adb814670a8542e9b26b95776 to finish bootstrapping
I20250626 20:07:23.503696 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d20b583c8a7e434090db8ab6a8e540bf to finish bootstrapping
I20250626 20:07:23.506129 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7ede26a2a4f04c20aff525120a238252 to finish bootstrapping
I20250626 20:07:23.508718 22650 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33892:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250626 20:07:23.508814 22650 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 20:07:23.509725 22903 tablet_service.cc:1468] Processing CreateTablet for tablet e86175be8a484f929461c14300171db7 (DEFAULT_TABLE table=TestAnotherTable [id=580ad80a478b4536acdca9eda05031c4]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 20:07:23.509725 22772 tablet_service.cc:1468] Processing CreateTablet for tablet e86175be8a484f929461c14300171db7 (DEFAULT_TABLE table=TestAnotherTable [id=580ad80a478b4536acdca9eda05031c4]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 20:07:23.509817 22772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e86175be8a484f929461c14300171db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.509817 22903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e86175be8a484f929461c14300171db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.510650 23034 tablet_service.cc:1468] Processing CreateTablet for tablet e86175be8a484f929461c14300171db7 (DEFAULT_TABLE table=TestAnotherTable [id=580ad80a478b4536acdca9eda05031c4]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 20:07:23.510737 23034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e86175be8a484f929461c14300171db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:23.510805 23115 tablet_bootstrap.cc:492] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf: Bootstrap starting.
I20250626 20:07:23.511160 23115 tablet_bootstrap.cc:654] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.511478 23114 tablet_bootstrap.cc:492] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776: Bootstrap starting.
I20250626 20:07:23.511684 23115 tablet_bootstrap.cc:492] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf: No bootstrap required, opened a new log
I20250626 20:07:23.511722 23115 ts_tablet_manager.cc:1397] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250626 20:07:23.511801 23118 tablet_bootstrap.cc:492] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252: Bootstrap starting.
I20250626 20:07:23.511842 23115 raft_consensus.cc:357] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.511904 23115 raft_consensus.cc:383] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.511925 23115 raft_consensus.cc:738] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20b583c8a7e434090db8ab6a8e540bf, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.511951 23114 tablet_bootstrap.cc:654] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.511978 23115 consensus_queue.cc:260] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.512054 23115 ts_tablet_manager.cc:1428] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.512147 23118 tablet_bootstrap.cc:654] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:23.512496 23114 tablet_bootstrap.cc:492] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776: No bootstrap required, opened a new log
I20250626 20:07:23.512539 23114 ts_tablet_manager.cc:1397] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:23.512616 23118 tablet_bootstrap.cc:492] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252: No bootstrap required, opened a new log
I20250626 20:07:23.512643 23118 ts_tablet_manager.cc:1397] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:23.512667 23114 raft_consensus.cc:357] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.512723 23114 raft_consensus.cc:383] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.512739 23114 raft_consensus.cc:738] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3ef21adb814670a8542e9b26b95776, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.512765 23118 raft_consensus.cc:357] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.512778 23114 consensus_queue.cc:260] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.512807 23118 raft_consensus.cc:383] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:23.512820 23118 raft_consensus.cc:738] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ede26a2a4f04c20aff525120a238252, State: Initialized, Role: FOLLOWER
I20250626 20:07:23.512852 23114 ts_tablet_manager.cc:1428] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.512852 23118 consensus_queue.cc:260] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.512925 23118 ts_tablet_manager.cc:1428] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:23.565274 23122 raft_consensus.cc:491] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:23.565320 23122 raft_consensus.cc:513] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.565399 23122 leader_election.cc:290] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c3ef21adb814670a8542e9b26b95776 (127.20.190.193:37073), d20b583c8a7e434090db8ab6a8e540bf (127.20.190.194:45509)
I20250626 20:07:23.565567 22923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e86175be8a484f929461c14300171db7" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20b583c8a7e434090db8ab6a8e540bf" is_pre_election: true
I20250626 20:07:23.565567 22792 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e86175be8a484f929461c14300171db7" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3ef21adb814670a8542e9b26b95776" is_pre_election: true
I20250626 20:07:23.565655 22923 raft_consensus.cc:2466] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 0.
I20250626 20:07:23.565655 22792 raft_consensus.cc:2466] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 0.
I20250626 20:07:23.565789 22988 leader_election.cc:304] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3ef21adb814670a8542e9b26b95776, 7ede26a2a4f04c20aff525120a238252; no voters:
I20250626 20:07:23.565882 23122 raft_consensus.cc:2802] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:23.565925 23122 raft_consensus.cc:491] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:23.565939 23122 raft_consensus.cc:3058] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.566372 23122 raft_consensus.cc:513] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.566469 23122 leader_election.cc:290] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 election: Requested vote from peers 2c3ef21adb814670a8542e9b26b95776 (127.20.190.193:37073), d20b583c8a7e434090db8ab6a8e540bf (127.20.190.194:45509)
I20250626 20:07:23.566642 22923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e86175be8a484f929461c14300171db7" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20b583c8a7e434090db8ab6a8e540bf"
I20250626 20:07:23.566653 22792 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e86175be8a484f929461c14300171db7" candidate_uuid: "7ede26a2a4f04c20aff525120a238252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3ef21adb814670a8542e9b26b95776"
I20250626 20:07:23.566699 22923 raft_consensus.cc:3058] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.566706 22792 raft_consensus.cc:3058] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:23.567265 22923 raft_consensus.cc:2466] T e86175be8a484f929461c14300171db7 P d20b583c8a7e434090db8ab6a8e540bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 1.
I20250626 20:07:23.567306 22792 raft_consensus.cc:2466] T e86175be8a484f929461c14300171db7 P 2c3ef21adb814670a8542e9b26b95776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ede26a2a4f04c20aff525120a238252 in term 1.
I20250626 20:07:23.567414 22990 leader_election.cc:304] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ede26a2a4f04c20aff525120a238252, d20b583c8a7e434090db8ab6a8e540bf; no voters:
I20250626 20:07:23.567528 23122 raft_consensus.cc:2802] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:23.567579 23122 raft_consensus.cc:695] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [term 1 LEADER]: Becoming Leader. State: Replica: 7ede26a2a4f04c20aff525120a238252, State: Running, Role: LEADER
I20250626 20:07:23.567633 23122 consensus_queue.cc:237] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } }
I20250626 20:07:23.568018 22649 catalog_manager.cc:5582] T e86175be8a484f929461c14300171db7 P 7ede26a2a4f04c20aff525120a238252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ede26a2a4f04c20aff525120a238252 (127.20.190.195). New cstate: current_term: 1 leader_uuid: "7ede26a2a4f04c20aff525120a238252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3ef21adb814670a8542e9b26b95776" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20b583c8a7e434090db8ab6a8e540bf" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ede26a2a4f04c20aff525120a238252" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 33615 } health_report { overall_health: HEALTHY } } }
W20250626 20:07:23.575286 22970 tablet.cc:2378] T 0259266badac49bf8560179b45ea0dd2 P d20b583c8a7e434090db8ab6a8e540bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:23.648026 23130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.648149 23130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.649330 23130 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 20:07:23.664343 23101 tablet.cc:2378] T 0259266badac49bf8560179b45ea0dd2 P 7ede26a2a4f04c20aff525120a238252: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:23.725127 23140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.725248 23140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.726477 23140 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 20:07:23.800778 23150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.800895 23150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.802006 23150 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 20:07:23.879384 23160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.879506 23160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.880679 23160 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 20:07:23.894326 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22710
I20250626 20:07:23.898542 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22841
I20250626 20:07:23.902300 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22972
I20250626 20:07:23.906294 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 22619
2025-06-26T20:07:23Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (927 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250626 20:07:23.925611 21243 test_util.cc:276] Using random seed: 471659625
I20250626 20:07:23.926066 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:23.926100 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:23.926116 21243 ts_itest-base.cc:117] 3 tablet servers
I20250626 20:07:23.926133 21243 ts_itest-base.cc:118] 3 replicas per TS
I20250626 20:07:23.926158 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:23Z Disabled control of system clock
I20250626 20:07:23.932403 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:40855
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:40855
--superuser_acl=no-such-user with env {}
W20250626 20:07:23.996321 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:23.996456 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:23.996484 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:23.997663 23177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:23.997696 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:23.997715 23177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:23.997733 23177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:23.998973 23177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:40855
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:23.999154 23177 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:23.999323 23177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:24.002126 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.002203 23177 server_base.cc:1048] running on GCE node
W20250626 20:07:24.002081 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.002076 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.002506 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:24.002714 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:24.003824 23177 hybrid_clock.cc:648] HybridClock initialized: now 1750968444003817 us; error 30 us; skew 500 ppm
I20250626 20:07:24.004755 23177 webserver.cc:469] Webserver started at http://127.20.190.254:42013/ using document root <none> and password file <none>
I20250626 20:07:24.004897 23177 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:24.004930 23177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:24.005018 23177 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:24.005671 23177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4eec8fa9f428494398339c6dc8aaa5ee"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.005987 23177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4eec8fa9f428494398339c6dc8aaa5ee"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.007074 23177 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:24.007570 23191 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.007699 23177 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.007757 23177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4eec8fa9f428494398339c6dc8aaa5ee"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.007803 23177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:24.022318 23177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:24.022547 23177 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:24.022644 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:24.025743 23177 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:40855
I20250626 20:07:24.025779 23243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:40855 every 8 connection(s)
I20250626 20:07:24.026011 23177 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:24.026552 23244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:24.028369 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee: Bootstrap starting.
I20250626 20:07:24.028834 23244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:24.029026 23244 log.cc:826] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:24.029479 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee: No bootstrap required, opened a new log
I20250626 20:07:24.030380 23244 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 40855 } }
I20250626 20:07:24.030469 23244 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:24.030496 23244 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec8fa9f428494398339c6dc8aaa5ee, State: Initialized, Role: FOLLOWER
I20250626 20:07:24.030625 23244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 40855 } }
I20250626 20:07:24.030683 23244 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:24.030712 23244 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:24.030748 23244 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:24.031155 23244 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 40855 } }
I20250626 20:07:24.031234 23244 leader_election.cc:304] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eec8fa9f428494398339c6dc8aaa5ee; no voters:
I20250626 20:07:24.031338 23244 leader_election.cc:290] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:24.031378 23247 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:24.031498 23247 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [term 1 LEADER]: Becoming Leader. State: Replica: 4eec8fa9f428494398339c6dc8aaa5ee, State: Running, Role: LEADER
I20250626 20:07:24.031512 23244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:24.031615 23247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 40855 } }
I20250626 20:07:24.031899 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 40855 } } }
I20250626 20:07:24.031917 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eec8fa9f428494398339c6dc8aaa5ee. Latest consensus state: current_term: 1 leader_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eec8fa9f428494398339c6dc8aaa5ee" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 40855 } } }
I20250626 20:07:24.032078 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:24.032018 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:24.032387 23257 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:24.032788 23257 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:24.034018 23257 catalog_manager.cc:1349] Generated new cluster ID: a7048683ab1043379be917a2beca8ac4
I20250626 20:07:24.034054 23257 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:24.035393 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23177
I20250626 20:07:24.035481 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:24.039747 23257 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:24.040326 23257 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:24.044945 23257 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4eec8fa9f428494398339c6dc8aaa5ee: Generated new TSK 0
I20250626 20:07:24.045066 23257 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:24.050189 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:40855
--builtin_ntp_servers=127.20.190.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:24.130899 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:24.131023 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:24.131052 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:24.132251 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:24.132299 23268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:24.133549 23268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:40855
--never_fsync=true
--heap_profile_path=/tmp/kudu.23268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:24.133739 23268 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:24.133924 23268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:24.137744 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.137768 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.137876 23268 server_base.cc:1048] running on GCE node
W20250626 20:07:24.138017 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.138152 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:24.138350 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:24.139501 23268 hybrid_clock.cc:648] HybridClock initialized: now 1750968444139484 us; error 28 us; skew 500 ppm
I20250626 20:07:24.140718 23268 webserver.cc:469] Webserver started at http://127.20.190.193:36475/ using document root <none> and password file <none>
I20250626 20:07:24.140911 23268 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:24.140960 23268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:24.141063 23268 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:24.141932 23268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.142249 23268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.143527 23268 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20250626 20:07:24.144186 23282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.144325 23268 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:24.144394 23268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.144476 23268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:24.184230 23268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:24.184725 23268 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:24.184851 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:24.185083 23268 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:24.185395 23268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:24.185438 23268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.185473 23268 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:24.185497 23268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.191893 23268 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:35775
I20250626 20:07:24.191937 23395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:35775 every 8 connection(s)
I20250626 20:07:24.192176 23268 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:24.196388 23396 heartbeater.cc:344] Connected to a master server at 127.20.190.254:40855
I20250626 20:07:24.196476 23396 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:24.196630 23396 heartbeater.cc:507] Master 127.20.190.254:40855 requested a full tablet report, sending...
I20250626 20:07:24.197016 23208 ts_manager.cc:194] Registered new tserver with Master: d2bb87a35c584d4f9d0d51ff7aa7e215 (127.20.190.193:35775)
I20250626 20:07:24.197021 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23268
I20250626 20:07:24.197079 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:24.197614 23208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:55533
I20250626 20:07:24.198060 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:40855
--builtin_ntp_servers=127.20.190.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:24.262970 23399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:24.263113 23399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:24.263140 23399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:24.264290 23399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:24.264343 23399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:24.265703 23399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:40855
--never_fsync=true
--heap_profile_path=/tmp/kudu.23399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:24.265937 23399 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:24.266129 23399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:24.269255 23407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.269268 23405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.269291 23404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.269416 23399 server_base.cc:1048] running on GCE node
I20250626 20:07:24.269564 23399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:24.269693 23399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:24.270807 23399 hybrid_clock.cc:648] HybridClock initialized: now 1750968444270798 us; error 23 us; skew 500 ppm
I20250626 20:07:24.271749 23399 webserver.cc:469] Webserver started at http://127.20.190.194:38333/ using document root <none> and password file <none>
I20250626 20:07:24.271898 23399 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:24.271929 23399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:24.272015 23399 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:24.272658 23399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e258b71770214a4ababeabb661497990"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.272891 23399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e258b71770214a4ababeabb661497990"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.273795 23399 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:24.274286 23413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.274407 23399 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.274468 23399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e258b71770214a4ababeabb661497990"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.274542 23399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:24.299533 23399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:24.299736 23399 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:24.299825 23399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:24.299985 23399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:24.300204 23399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:24.300231 23399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.300259 23399 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:24.300280 23399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.304898 23399 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:45247
I20250626 20:07:24.304950 23526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:45247 every 8 connection(s)
I20250626 20:07:24.305159 23399 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:24.308842 23527 heartbeater.cc:344] Connected to a master server at 127.20.190.254:40855
I20250626 20:07:24.308919 23527 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:24.309064 23527 heartbeater.cc:507] Master 127.20.190.254:40855 requested a full tablet report, sending...
I20250626 20:07:24.309322 23207 ts_manager.cc:194] Registered new tserver with Master: e258b71770214a4ababeabb661497990 (127.20.190.194:45247)
I20250626 20:07:24.309633 23207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:45603
I20250626 20:07:24.311816 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23399
I20250626 20:07:24.311882 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 20:07:24.312659 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:40855
--builtin_ntp_servers=127.20.190.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:24.376688 23530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:24.376834 23530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:24.376863 23530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:24.378019 23530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:24.378067 23530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:24.379230 23530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:40855
--never_fsync=true
--heap_profile_path=/tmp/kudu.23530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:24.379422 23530 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:24.379594 23530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:24.382424 23538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.382429 23536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.382437 23535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.382611 23530 server_base.cc:1048] running on GCE node
I20250626 20:07:24.382735 23530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:24.382918 23530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:24.384025 23530 hybrid_clock.cc:648] HybridClock initialized: now 1750968444384011 us; error 26 us; skew 500 ppm
I20250626 20:07:24.385031 23530 webserver.cc:469] Webserver started at http://127.20.190.195:38781/ using document root <none> and password file <none>
I20250626 20:07:24.385190 23530 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:24.385227 23530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:24.385317 23530 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:24.386014 23530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "17ac88735f614c1f8d178f48f8ebc8f3"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.386276 23530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "17ac88735f614c1f8d178f48f8ebc8f3"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.387349 23530 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:24.387857 23544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.387985 23530 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:24.388042 23530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "17ac88735f614c1f8d178f48f8ebc8f3"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:24.388088 23530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:24.410781 23530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:24.411005 23530 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:24.411108 23530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:24.411296 23530 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:24.411512 23530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:24.411549 23530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.411576 23530 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:24.411626 23530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:24.416400 23530 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:41215
I20250626 20:07:24.416458 23657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:41215 every 8 connection(s)
I20250626 20:07:24.416666 23530 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:24.420498 23658 heartbeater.cc:344] Connected to a master server at 127.20.190.254:40855
I20250626 20:07:24.420567 23658 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:24.420715 23658 heartbeater.cc:507] Master 127.20.190.254:40855 requested a full tablet report, sending...
I20250626 20:07:24.420972 23207 ts_manager.cc:194] Registered new tserver with Master: 17ac88735f614c1f8d178f48f8ebc8f3 (127.20.190.195:41215)
I20250626 20:07:24.421258 23207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:47739
I20250626 20:07:24.426031 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23530
I20250626 20:07:24.426115 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 20:07:24.426961 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:24.431133 23207 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48156:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 20:07:24.432129 23207 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 20:07:24.436825 23330 tablet_service.cc:1468] Processing CreateTablet for tablet 0bb70971b6cf4baf9d7abb1a5ce287ae (DEFAULT_TABLE table=TestTable [id=aed0f075283c4594b1a29075a1d1d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:24.437114 23330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb70971b6cf4baf9d7abb1a5ce287ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:24.437284 23592 tablet_service.cc:1468] Processing CreateTablet for tablet 0bb70971b6cf4baf9d7abb1a5ce287ae (DEFAULT_TABLE table=TestTable [id=aed0f075283c4594b1a29075a1d1d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:24.437492 23592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb70971b6cf4baf9d7abb1a5ce287ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:24.439585 23672 tablet_bootstrap.cc:492] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: Bootstrap starting.
I20250626 20:07:24.439656 23673 tablet_bootstrap.cc:492] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: Bootstrap starting.
I20250626 20:07:24.440055 23461 tablet_service.cc:1468] Processing CreateTablet for tablet 0bb70971b6cf4baf9d7abb1a5ce287ae (DEFAULT_TABLE table=TestTable [id=aed0f075283c4594b1a29075a1d1d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:24.440279 23461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb70971b6cf4baf9d7abb1a5ce287ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:24.440279 23672 tablet_bootstrap.cc:654] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:24.440279 23673 tablet_bootstrap.cc:654] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:24.440554 23673 log.cc:826] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:24.440562 23672 log.cc:826] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:24.441150 23672 tablet_bootstrap.cc:492] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: No bootstrap required, opened a new log
I20250626 20:07:24.441216 23672 ts_tablet_manager.cc:1397] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:24.441329 23673 tablet_bootstrap.cc:492] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: No bootstrap required, opened a new log
I20250626 20:07:24.441401 23673 ts_tablet_manager.cc:1397] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:24.442394 23676 tablet_bootstrap.cc:492] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: Bootstrap starting.
I20250626 20:07:24.442744 23672 raft_consensus.cc:357] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.442734 23673 raft_consensus.cc:357] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.442860 23673 raft_consensus.cc:383] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:24.442860 23672 raft_consensus.cc:383] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:24.442888 23673 raft_consensus.cc:738] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ac88735f614c1f8d178f48f8ebc8f3, State: Initialized, Role: FOLLOWER
I20250626 20:07:24.442888 23672 raft_consensus.cc:738] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bb87a35c584d4f9d0d51ff7aa7e215, State: Initialized, Role: FOLLOWER
I20250626 20:07:24.442943 23676 tablet_bootstrap.cc:654] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:24.442989 23672 consensus_queue.cc:260] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.442992 23673 consensus_queue.cc:260] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.443194 23676 log.cc:826] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:24.443197 23672 ts_tablet_manager.cc:1428] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:24.443197 23673 ts_tablet_manager.cc:1428] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:24.443288 23658 heartbeater.cc:499] Master 127.20.190.254:40855 was elected leader, sending a full tablet report...
I20250626 20:07:24.443288 23396 heartbeater.cc:499] Master 127.20.190.254:40855 was elected leader, sending a full tablet report...
I20250626 20:07:24.443921 23676 tablet_bootstrap.cc:492] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: No bootstrap required, opened a new log
I20250626 20:07:24.443980 23676 ts_tablet_manager.cc:1397] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:24.445153 23676 raft_consensus.cc:357] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.445267 23676 raft_consensus.cc:383] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:24.445289 23676 raft_consensus.cc:738] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e258b71770214a4ababeabb661497990, State: Initialized, Role: FOLLOWER
I20250626 20:07:24.445365 23676 consensus_queue.cc:260] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.445547 23676 ts_tablet_manager.cc:1428] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:24.445582 23527 heartbeater.cc:499] Master 127.20.190.254:40855 was elected leader, sending a full tablet report...
W20250626 20:07:24.555549 23528 tablet.cc:2378] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:24.667191 23659 tablet.cc:2378] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:24.692804 23397 tablet.cc:2378] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:24.755563 23677 raft_consensus.cc:491] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:24.755661 23677 raft_consensus.cc:513] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.755964 23677 leader_election.cc:290] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2bb87a35c584d4f9d0d51ff7aa7e215 (127.20.190.193:35775), e258b71770214a4ababeabb661497990 (127.20.190.194:45247)
I20250626 20:07:24.758504 23350 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bb70971b6cf4baf9d7abb1a5ce287ae" candidate_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" is_pre_election: true
I20250626 20:07:24.758664 23350 raft_consensus.cc:2466] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ac88735f614c1f8d178f48f8ebc8f3 in term 0.
I20250626 20:07:24.758874 23548 leader_election.cc:304] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ac88735f614c1f8d178f48f8ebc8f3, d2bb87a35c584d4f9d0d51ff7aa7e215; no voters:
I20250626 20:07:24.758963 23481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bb70971b6cf4baf9d7abb1a5ce287ae" candidate_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e258b71770214a4ababeabb661497990" is_pre_election: true
I20250626 20:07:24.759011 23677 raft_consensus.cc:2802] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:24.759068 23677 raft_consensus.cc:491] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:24.759097 23677 raft_consensus.cc:3058] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:24.759096 23481 raft_consensus.cc:2466] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ac88735f614c1f8d178f48f8ebc8f3 in term 0.
I20250626 20:07:24.759693 23677 raft_consensus.cc:513] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.759791 23677 leader_election.cc:290] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [CANDIDATE]: Term 1 election: Requested vote from peers d2bb87a35c584d4f9d0d51ff7aa7e215 (127.20.190.193:35775), e258b71770214a4ababeabb661497990 (127.20.190.194:45247)
I20250626 20:07:24.759925 23350 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bb70971b6cf4baf9d7abb1a5ce287ae" candidate_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215"
I20250626 20:07:24.759924 23481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bb70971b6cf4baf9d7abb1a5ce287ae" candidate_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e258b71770214a4ababeabb661497990"
I20250626 20:07:24.759984 23350 raft_consensus.cc:3058] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:24.759989 23481 raft_consensus.cc:3058] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:24.760492 23350 raft_consensus.cc:2466] T 0bb70971b6cf4baf9d7abb1a5ce287ae P d2bb87a35c584d4f9d0d51ff7aa7e215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ac88735f614c1f8d178f48f8ebc8f3 in term 1.
I20250626 20:07:24.760628 23548 leader_election.cc:304] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ac88735f614c1f8d178f48f8ebc8f3, d2bb87a35c584d4f9d0d51ff7aa7e215; no voters:
I20250626 20:07:24.760701 23677 raft_consensus.cc:2802] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:24.760818 23677 raft_consensus.cc:695] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [term 1 LEADER]: Becoming Leader. State: Replica: 17ac88735f614c1f8d178f48f8ebc8f3, State: Running, Role: LEADER
I20250626 20:07:24.760834 23481 raft_consensus.cc:2466] T 0bb70971b6cf4baf9d7abb1a5ce287ae P e258b71770214a4ababeabb661497990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ac88735f614c1f8d178f48f8ebc8f3 in term 1.
I20250626 20:07:24.760915 23677 consensus_queue.cc:237] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } }
I20250626 20:07:24.761570 23207 catalog_manager.cc:5582] T 0bb70971b6cf4baf9d7abb1a5ce287ae P 17ac88735f614c1f8d178f48f8ebc8f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17ac88735f614c1f8d178f48f8ebc8f3 (127.20.190.195). New cstate: current_term: 1 leader_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ac88735f614c1f8d178f48f8ebc8f3" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 41215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2bb87a35c584d4f9d0d51ff7aa7e215" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e258b71770214a4ababeabb661497990" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45247 } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:24.784390 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:24.784780 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d2bb87a35c584d4f9d0d51ff7aa7e215 to finish bootstrapping
I20250626 20:07:24.792562 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e258b71770214a4ababeabb661497990 to finish bootstrapping
I20250626 20:07:24.794947 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 17ac88735f614c1f8d178f48f8ebc8f3 to finish bootstrapping
W20250626 20:07:24.884140 23207 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:48178
I20250626 20:07:24.888957 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23268
I20250626 20:07:24.894737 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23399
I20250626 20:07:24.899477 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23530
I20250626 20:07:24.903460 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23177
2025-06-26T20:07:24Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (996 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250626 20:07:24.921813 21243 test_util.cc:276] Using random seed: 472655827
I20250626 20:07:24.922194 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:24.922225 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:24.922238 21243 ts_itest-base.cc:117] 3 tablet servers
I20250626 20:07:24.922253 21243 ts_itest-base.cc:118] 3 replicas per TS
I20250626 20:07:24.922274 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:24Z Disabled control of system clock
I20250626 20:07:24.927966 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:46165
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:46165 with env {}
W20250626 20:07:24.991576 23707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:24.991700 23707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:24.991726 23707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:24.992861 23707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:24.992897 23707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:24.992933 23707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:24.992954 23707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:24.994155 23707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:46165
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:24.994340 23707 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:24.994506 23707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:24.997434 23715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:24.997468 23712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.997515 23707 server_base.cc:1048] running on GCE node
W20250626 20:07:24.997443 23713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:24.997778 23707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:24.997953 23707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:24.999081 23707 hybrid_clock.cc:648] HybridClock initialized: now 1750968444999066 us; error 28 us; skew 500 ppm
I20250626 20:07:24.999982 23707 webserver.cc:469] Webserver started at http://127.20.190.254:42993/ using document root <none> and password file <none>
I20250626 20:07:25.000125 23707 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:25.000162 23707 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:25.000250 23707 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:25.000900 23707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "677637a36d1e405bb222c2c8bea54649"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:25.001133 23707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "677637a36d1e405bb222c2c8bea54649"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:25.002035 23707 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:25.002542 23721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.002663 23707 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.002722 23707 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "677637a36d1e405bb222c2c8bea54649"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:25.002769 23707 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:25.010103 23707 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:25.010313 23707 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:25.010402 23707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:25.013464 23707 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:46165
I20250626 20:07:25.013494 23773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:46165 every 8 connection(s)
I20250626 20:07:25.013808 23707 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:25.014246 23774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:25.016059 23774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap starting.
I20250626 20:07:25.016561 23774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:25.016783 23774 log.cc:826] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:25.017266 23774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: No bootstrap required, opened a new log
I20250626 20:07:25.018144 23774 raft_consensus.cc:357] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.018235 23774 raft_consensus.cc:383] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:25.018263 23774 raft_consensus.cc:738] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Initialized, Role: FOLLOWER
I20250626 20:07:25.018355 23774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.018415 23774 raft_consensus.cc:397] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:25.018443 23774 raft_consensus.cc:491] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:25.018479 23774 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:25.018906 23774 raft_consensus.cc:513] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.018987 23774 leader_election.cc:304] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677637a36d1e405bb222c2c8bea54649; no voters:
I20250626 20:07:25.019101 23774 leader_election.cc:290] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:25.019145 23777 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:25.019255 23777 raft_consensus.cc:695] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 1 LEADER]: Becoming Leader. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Running, Role: LEADER
I20250626 20:07:25.019271 23774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:25.019353 23777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.019630 23778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:25.019645 23779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677637a36d1e405bb222c2c8bea54649. Latest consensus state: current_term: 1 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:25.019798 23779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:25.019786 23778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:25.020095 23785 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:25.020424 23785 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:25.020789 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23707
I20250626 20:07:25.020877 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:25.021512 23785 catalog_manager.cc:1349] Generated new cluster ID: f44250e2f8bb460dae5e937c802164a9
I20250626 20:07:25.021646 23785 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:25.034021 23785 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:25.034394 23785 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:25.037798 23785 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Generated new TSK 0
I20250626 20:07:25.037906 23785 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:25.042182 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:46165
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:25.106264 23798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:25.106395 23798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:25.106413 23798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:25.107637 23798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:25.107681 23798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:25.108844 23798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.23798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:25.109006 23798 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:25.109176 23798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:25.112144 23804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:25.112188 23803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:25.112268 23798 server_base.cc:1048] running on GCE node
W20250626 20:07:25.112351 23806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:25.112500 23798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:25.112663 23798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:25.113793 23798 hybrid_clock.cc:648] HybridClock initialized: now 1750968445113786 us; error 23 us; skew 500 ppm
I20250626 20:07:25.114781 23798 webserver.cc:469] Webserver started at http://127.20.190.193:37031/ using document root <none> and password file <none>
I20250626 20:07:25.114964 23798 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:25.115006 23798 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:25.115090 23798 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:25.115767 23798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "77208ca02c274a27a7ed27c2fed0ac69"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.116030 23798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "77208ca02c274a27a7ed27c2fed0ac69"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.117137 23798 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:25.117686 23812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.117827 23798 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:25.117883 23798 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "77208ca02c274a27a7ed27c2fed0ac69"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.117933 23798 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:25.132797 23798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:25.133006 23798 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:25.133098 23798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:25.133251 23798 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:25.133512 23798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:25.133544 23798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.133569 23798 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:25.133594 23798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.138873 23798 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:37699
I20250626 20:07:25.138913 23925 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:37699 every 8 connection(s)
I20250626 20:07:25.139159 23798 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:25.143508 23926 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:25.143599 23926 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:25.143846 23926 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:25.144284 23738 ts_manager.cc:194] Registered new tserver with Master: 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699)
I20250626 20:07:25.145151 23738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:46953
I20250626 20:07:25.145417 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23798
I20250626 20:07:25.145500 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:25.146471 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:46165
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:25.211113 23929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:25.211228 23929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:25.211244 23929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:25.212363 23929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:25.212406 23929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:25.213572 23929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.23929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:25.213709 23929 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:25.213876 23929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:25.216840 23935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:25.216920 23937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:25.216852 23934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:25.217089 23929 server_base.cc:1048] running on GCE node
I20250626 20:07:25.217213 23929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:25.217342 23929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:25.218451 23929 hybrid_clock.cc:648] HybridClock initialized: now 1750968445218445 us; error 23 us; skew 500 ppm
I20250626 20:07:25.219338 23929 webserver.cc:469] Webserver started at http://127.20.190.194:45971/ using document root <none> and password file <none>
I20250626 20:07:25.219486 23929 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:25.219519 23929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:25.219614 23929 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:25.220273 23929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fbfca0cc6a254dab9b1efa2dcf374770"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.220494 23929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fbfca0cc6a254dab9b1efa2dcf374770"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.221451 23929 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:25.221936 23943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.222056 23929 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.222108 23929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fbfca0cc6a254dab9b1efa2dcf374770"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.222155 23929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:25.233058 23929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:25.233256 23929 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:25.233343 23929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:25.233490 23929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:25.233744 23929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:25.233773 23929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.233800 23929 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:25.233827 23929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.239001 23929 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:45511
I20250626 20:07:25.239043 24056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:45511 every 8 connection(s)
I20250626 20:07:25.239262 23929 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:25.239889 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 23929
I20250626 20:07:25.239979 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 20:07:25.241120 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:46165
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:25.243371 24057 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:25.243441 24057 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:25.243598 24057 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:25.243921 23738 ts_manager.cc:194] Registered new tserver with Master: fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:25.244246 23738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:33957
W20250626 20:07:25.309181 24060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:25.309320 24060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:25.309346 24060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:25.310593 24060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:25.310642 24060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:25.311868 24060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.24060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:25.312072 24060 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:25.312243 24060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:25.315196 24066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:25.315232 24068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:25.315330 24060 server_base.cc:1048] running on GCE node
W20250626 20:07:25.315367 24065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:25.315548 24060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:25.315758 24060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:25.316885 24060 hybrid_clock.cc:648] HybridClock initialized: now 1750968445316871 us; error 32 us; skew 500 ppm
I20250626 20:07:25.317907 24060 webserver.cc:469] Webserver started at http://127.20.190.195:41437/ using document root <none> and password file <none>
I20250626 20:07:25.318064 24060 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:25.318100 24060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:25.318217 24060 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:25.318939 24060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "672299fad8844abd851a2a853328394c"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.319185 24060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "672299fad8844abd851a2a853328394c"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.320135 24060 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:25.320726 24074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.320863 24060 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:25.320919 24060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "672299fad8844abd851a2a853328394c"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:25.320971 24060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:25.341066 24060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:25.341280 24060 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:25.341377 24060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:25.341547 24060 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:25.341775 24060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:25.341802 24060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.341830 24060 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:25.341854 24060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.346812 24060 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:42075
I20250626 20:07:25.346869 24187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:42075 every 8 connection(s)
I20250626 20:07:25.347088 24060 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:25.350288 24188 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:25.350373 24188 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:25.350538 24188 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:25.350829 23738 ts_manager.cc:194] Registered new tserver with Master: 672299fad8844abd851a2a853328394c (127.20.190.195:42075)
I20250626 20:07:25.351150 23738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:60401
I20250626 20:07:25.355052 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 24060
I20250626 20:07:25.355123 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 20:07:25.355914 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:25.360773 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:25.360826 21243 test_util.cc:276] Using random seed: 473094842
I20250626 20:07:25.365598 23738 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45230:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 20:07:25.370178 23991 tablet_service.cc:1468] Processing CreateTablet for tablet d7dfd28923b74a75b1f4df0aa6caab31 (DEFAULT_TABLE table=TestTable [id=05408b708c0f4ca9b03e48d88fd72503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:25.370425 23991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7dfd28923b74a75b1f4df0aa6caab31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:25.372074 24203 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap starting.
I20250626 20:07:25.372506 24203 tablet_bootstrap.cc:654] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:25.372758 24203 log.cc:826] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:25.373268 24203 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: No bootstrap required, opened a new log
I20250626 20:07:25.373311 24203 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:25.374172 24203 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:25.374264 24203 raft_consensus.cc:383] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:25.374281 24203 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Initialized, Role: FOLLOWER
I20250626 20:07:25.374356 24203 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:25.374409 24203 raft_consensus.cc:397] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:25.374430 24203 raft_consensus.cc:491] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:25.374469 24203 raft_consensus.cc:3058] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:25.374908 24203 raft_consensus.cc:513] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:25.374986 24203 leader_election.cc:304] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbfca0cc6a254dab9b1efa2dcf374770; no voters:
I20250626 20:07:25.375108 24203 leader_election.cc:290] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:25.375149 24205 raft_consensus.cc:2802] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:25.375280 24205 raft_consensus.cc:695] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 LEADER]: Becoming Leader. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Running, Role: LEADER
I20250626 20:07:25.375270 24057 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:25.375259 24203 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:25.375371 24205 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:25.376084 23738 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194). New cstate: current_term: 1 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:25.400676 21243 test_util.cc:276] Using random seed: 473134688
I20250626 20:07:25.404174 23738 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45244:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 20:07:25.408113 24122 tablet_service.cc:1468] Processing CreateTablet for tablet 30adbf13379941e2b1c4857ae476b139 (DEFAULT_TABLE table=TestTable1 [id=27d0bfb856c64f14a03d11be879562c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:25.408320 24122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30adbf13379941e2b1c4857ae476b139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:25.409880 24225 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap starting.
I20250626 20:07:25.410329 24225 tablet_bootstrap.cc:654] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:25.410562 24225 log.cc:826] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:25.411083 24225 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: No bootstrap required, opened a new log
I20250626 20:07:25.411130 24225 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250626 20:07:25.411948 24225 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:25.412035 24225 raft_consensus.cc:383] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:25.412052 24225 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672299fad8844abd851a2a853328394c, State: Initialized, Role: FOLLOWER
I20250626 20:07:25.412107 24225 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:25.412144 24225 raft_consensus.cc:397] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:25.412160 24225 raft_consensus.cc:491] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:25.412186 24225 raft_consensus.cc:3058] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:25.412571 24225 raft_consensus.cc:513] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:25.412642 24225 leader_election.cc:304] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672299fad8844abd851a2a853328394c; no voters:
I20250626 20:07:25.412732 24225 leader_election.cc:290] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:25.412775 24227 raft_consensus.cc:2802] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:25.412864 24225 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:25.412879 24188 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:25.412914 24227 raft_consensus.cc:695] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 LEADER]: Becoming Leader. State: Replica: 672299fad8844abd851a2a853328394c, State: Running, Role: LEADER
I20250626 20:07:25.412994 24227 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:25.413491 23738 catalog_manager.cc:5582] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c reported cstate change: term changed from 0 to 1, leader changed from <none> to 672299fad8844abd851a2a853328394c (127.20.190.195). New cstate: current_term: 1 leader_uuid: "672299fad8844abd851a2a853328394c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:25.438181 21243 test_util.cc:276] Using random seed: 473172190
I20250626 20:07:25.441658 23738 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45256:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 20:07:25.445334 23860 tablet_service.cc:1468] Processing CreateTablet for tablet 43cae63b9def4f79b9e1a3d28751e7b4 (DEFAULT_TABLE table=TestTable2 [id=ecc6bf3164db4166b6f547db186842fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:25.445541 23860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43cae63b9def4f79b9e1a3d28751e7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:25.447353 24247 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:25.447809 24247 tablet_bootstrap.cc:654] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:25.448012 24247 log.cc:826] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:25.448524 24247 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: No bootstrap required, opened a new log
I20250626 20:07:25.448575 24247 ts_tablet_manager.cc:1397] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:25.449568 24247 raft_consensus.cc:357] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:25.449658 24247 raft_consensus.cc:383] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:25.449676 24247 raft_consensus.cc:738] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: FOLLOWER
I20250626 20:07:25.449750 24247 consensus_queue.cc:260] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:25.449807 24247 raft_consensus.cc:397] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:25.449834 24247 raft_consensus.cc:491] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:25.449873 24247 raft_consensus.cc:3058] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:25.450273 24247 raft_consensus.cc:513] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:25.450352 24247 leader_election.cc:304] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77208ca02c274a27a7ed27c2fed0ac69; no voters:
I20250626 20:07:25.450469 24247 leader_election.cc:290] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:25.450542 24249 raft_consensus.cc:2802] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:25.450639 24247 ts_tablet_manager.cc:1428] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:25.450659 23926 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:25.450670 24249 raft_consensus.cc:695] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 LEADER]: Becoming Leader. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Running, Role: LEADER
I20250626 20:07:25.450793 24249 consensus_queue.cc:237] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:25.451318 23738 catalog_manager.cc:5582] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193). New cstate: current_term: 1 leader_uuid: "77208ca02c274a27a7ed27c2fed0ac69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:25.465544 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23707
I20250626 20:07:25.829124 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23798
I20250626 20:07:25.833180 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 23929
I20250626 20:07:25.837304 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 24060
I20250626 20:07:25.841306 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:46165
--webserver_interface=127.20.190.254
--webserver_port=42993
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:46165 with env {}
W20250626 20:07:25.909082 24306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:25.909214 24306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:25.909230 24306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:25.910485 24306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:25.910550 24306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:25.910583 24306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:25.910594 24306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:25.911827 24306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:46165
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=42993
--never_fsync=true
--heap_profile_path=/tmp/kudu.24306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:25.911986 24306 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:25.912128 24306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:25.914921 24312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:25.914929 24311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:25.915038 24314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:25.915148 24306 server_base.cc:1048] running on GCE node
I20250626 20:07:25.915382 24306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:25.915550 24306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:25.916673 24306 hybrid_clock.cc:648] HybridClock initialized: now 1750968445916669 us; error 37 us; skew 500 ppm
I20250626 20:07:25.917711 24306 webserver.cc:469] Webserver started at http://127.20.190.254:42993/ using document root <none> and password file <none>
I20250626 20:07:25.917879 24306 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:25.917919 24306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:25.918998 24306 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 20:07:25.919514 24320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:25.919657 24306 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:25.919716 24306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "677637a36d1e405bb222c2c8bea54649"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:25.919916 24306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:25.936856 24306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:25.937110 24306 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:25.937203 24306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:25.940506 24306 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:46165
I20250626 20:07:25.940539 24372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:46165 every 8 connection(s)
I20250626 20:07:25.940784 24306 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:25.941623 24373 sys_catalog.cc:263] Verifying existing consensus state
I20250626 20:07:25.942103 24373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap starting.
I20250626 20:07:25.943262 24373 log.cc:826] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:25.944689 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 24306
I20250626 20:07:25.944900 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:37699
--local_ip_for_outbound_sockets=127.20.190.193
--tserver_master_addrs=127.20.190.254:46165
--webserver_port=37031
--webserver_interface=127.20.190.193
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:25.945236 24373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:25.945462 24373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap complete.
I20250626 20:07:25.946704 24373 raft_consensus.cc:357] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.946879 24373 raft_consensus.cc:738] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Initialized, Role: FOLLOWER
I20250626 20:07:25.947008 24373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.947073 24373 raft_consensus.cc:397] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:25.947115 24373 raft_consensus.cc:491] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:25.947156 24373 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:25.947939 24373 raft_consensus.cc:513] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.948027 24373 leader_election.cc:304] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677637a36d1e405bb222c2c8bea54649; no voters:
I20250626 20:07:25.948135 24373 leader_election.cc:290] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 20:07:25.948179 24377 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 20:07:25.948297 24377 raft_consensus.cc:695] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 LEADER]: Becoming Leader. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Running, Role: LEADER
I20250626 20:07:25.948328 24373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:25.948410 24377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:25.948700 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:25.948709 24379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677637a36d1e405bb222c2c8bea54649. Latest consensus state: current_term: 3 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:25.948845 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:25.948851 24379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:25.949199 24382 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:25.949637 24382 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=4e861185dbcd47f293f7a44305abba55]
I20250626 20:07:25.949772 24382 catalog_manager.cc:671] Loaded metadata for table TestTable [id=c17a2779fbb145fda0c935b764824b02]
I20250626 20:07:25.949827 24382 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=ecc6bf3164db4166b6f547db186842fa]
I20250626 20:07:25.949965 24382 tablet_loader.cc:96] loaded metadata for tablet 30adbf13379941e2b1c4857ae476b139 (table TestTable1 [id=4e861185dbcd47f293f7a44305abba55])
I20250626 20:07:25.950059 24382 tablet_loader.cc:96] loaded metadata for tablet 43cae63b9def4f79b9e1a3d28751e7b4 (table TestTable2 [id=ecc6bf3164db4166b6f547db186842fa])
I20250626 20:07:25.950098 24382 tablet_loader.cc:96] loaded metadata for tablet d7dfd28923b74a75b1f4df0aa6caab31 (table TestTable [id=c17a2779fbb145fda0c935b764824b02])
I20250626 20:07:25.950126 24382 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:25.950240 24382 catalog_manager.cc:1261] Loaded cluster ID: f44250e2f8bb460dae5e937c802164a9
I20250626 20:07:25.950273 24382 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:25.951404 24382 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:25.951562 24382 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Loaded TSK: 0
I20250626 20:07:25.951742 24382 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 20:07:26.011091 24376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:26.011209 24376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:26.011235 24376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:26.012356 24376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:26.012404 24376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:26.013571 24376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:37699
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=37031
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.24376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:26.013751 24376 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:26.013934 24376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:26.017341 24400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:26.017395 24402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:26.017554 24376 server_base.cc:1048] running on GCE node
W20250626 20:07:26.017575 24399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:26.017748 24376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:26.017920 24376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:26.019039 24376 hybrid_clock.cc:648] HybridClock initialized: now 1750968446019025 us; error 27 us; skew 500 ppm
I20250626 20:07:26.019935 24376 webserver.cc:469] Webserver started at http://127.20.190.193:37031/ using document root <none> and password file <none>
I20250626 20:07:26.020088 24376 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:26.020138 24376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:26.021056 24376 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:26.021538 24408 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.021683 24376 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.021739 24376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "77208ca02c274a27a7ed27c2fed0ac69"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:26.021912 24376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:26.035229 24376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:26.035425 24376 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:26.035514 24376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:26.035676 24376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:26.036067 24415 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250626 20:07:26.036772 24376 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 20:07:26.036877 24376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:26.036957 24376 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 20:07:26.037402 24376 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 20:07:26.037487 24376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.037513 24415 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:26.042052 24376 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:37699
I20250626 20:07:26.042368 24376 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:26.043884 24522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:37699 every 8 connection(s)
I20250626 20:07:26.046838 24415 log.cc:826] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:26.048590 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 24376
I20250626 20:07:26.048846 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:45511
--local_ip_for_outbound_sockets=127.20.190.194
--tserver_master_addrs=127.20.190.254:46165
--webserver_port=45971
--webserver_interface=127.20.190.194
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:26.051187 24523 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:26.051263 24523 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:26.051410 24523 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:26.051991 24337 ts_manager.cc:194] Registered new tserver with Master: 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699)
I20250626 20:07:26.052611 24337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:50551
I20250626 20:07:26.052670 24415 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.052935 24415 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap complete.
I20250626 20:07:26.053902 24415 ts_tablet_manager.cc:1397] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20250626 20:07:26.054898 24415 raft_consensus.cc:357] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:26.055106 24415 raft_consensus.cc:738] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: FOLLOWER
I20250626 20:07:26.055195 24415 consensus_queue.cc:260] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:26.055250 24415 raft_consensus.cc:397] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:26.055285 24415 raft_consensus.cc:491] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:26.055321 24415 raft_consensus.cc:3058] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:26.056305 24415 raft_consensus.cc:513] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:26.056408 24415 leader_election.cc:304] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77208ca02c274a27a7ed27c2fed0ac69; no voters:
I20250626 20:07:26.056552 24415 leader_election.cc:290] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 20:07:26.056602 24529 raft_consensus.cc:2802] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:26.056740 24415 ts_tablet_manager.cc:1428] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20250626 20:07:26.056788 24523 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:26.056747 24529 raft_consensus.cc:695] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEADER]: Becoming Leader. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Running, Role: LEADER
I20250626 20:07:26.057066 24529 consensus_queue.cc:237] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:26.057600 24337 catalog_manager.cc:5582] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "77208ca02c274a27a7ed27c2fed0ac69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } health_report { overall_health: HEALTHY } } }
W20250626 20:07:26.121804 24528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:26.121923 24528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:26.121939 24528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:26.123150 24528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:26.123204 24528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:26.124433 24528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:45511
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=45971
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.24528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:26.124625 24528 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:26.124822 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:26.128034 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:26.128021 24539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:26.128044 24540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:26.128181 24528 server_base.cc:1048] running on GCE node
I20250626 20:07:26.128317 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:26.128479 24528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:26.129586 24528 hybrid_clock.cc:648] HybridClock initialized: now 1750968446129584 us; error 27 us; skew 500 ppm
I20250626 20:07:26.130465 24528 webserver.cc:469] Webserver started at http://127.20.190.194:45971/ using document root <none> and password file <none>
I20250626 20:07:26.130625 24528 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:26.130668 24528 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:26.131582 24528 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:26.132037 24548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.132170 24528 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.132223 24528 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fbfca0cc6a254dab9b1efa2dcf374770"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:26.132381 24528 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:26.157266 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:26.157536 24528 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:26.157618 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:26.157754 24528 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:26.158073 24555 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250626 20:07:26.158715 24528 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 20:07:26.158748 24528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:26.158767 24528 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 20:07:26.159159 24528 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 20:07:26.159184 24528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.159230 24555 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap starting.
I20250626 20:07:26.164122 24528 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:45511
I20250626 20:07:26.164201 24662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:45511 every 8 connection(s)
I20250626 20:07:26.164412 24528 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:26.168493 24663 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:26.168514 24555 log.cc:826] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:26.168591 24663 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:26.168756 24663 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:26.169155 24337 ts_manager.cc:194] Registered new tserver with Master: fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:26.169482 24337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:43591
I20250626 20:07:26.170050 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 24528
I20250626 20:07:26.170249 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:42075
--local_ip_for_outbound_sockets=127.20.190.195
--tserver_master_addrs=127.20.190.254:46165
--webserver_port=41437
--webserver_interface=127.20.190.195
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:26.173882 24555 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.174119 24555 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap complete.
I20250626 20:07:26.175033 24555 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20250626 20:07:26.176182 24555 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:26.176478 24555 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Initialized, Role: FOLLOWER
I20250626 20:07:26.176586 24555 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:26.176657 24555 raft_consensus.cc:397] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:26.176705 24555 raft_consensus.cc:491] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:26.176759 24555 raft_consensus.cc:3058] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:26.177848 24555 raft_consensus.cc:513] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:26.177949 24555 leader_election.cc:304] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbfca0cc6a254dab9b1efa2dcf374770; no voters:
I20250626 20:07:26.178085 24555 leader_election.cc:290] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 20:07:26.178120 24668 raft_consensus.cc:2802] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:26.178234 24668 raft_consensus.cc:695] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEADER]: Becoming Leader. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Running, Role: LEADER
I20250626 20:07:26.178251 24555 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250626 20:07:26.178318 24668 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:26.178371 24663 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:26.178802 24337 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: term changed from 0 to 2, leader changed from <none> to fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194), VOTER fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) added. New cstate: current_term: 2 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:26.182827 24617 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 32, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:26.183146 24668 raft_consensus.cc:2953] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index -1 to 34, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) added. New config: { opid_index: 34 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } }
I20250626 20:07:26.183936 24323 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 20:07:26.184037 24337 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: config changed from index -1 to 34, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) added. New cstate: current_term: 2 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: 34 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 20:07:26.184588 24337 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 with cas_config_opid_index 34: no extra replica candidate found for tablet d7dfd28923b74a75b1f4df0aa6caab31 (table TestTable [id=c17a2779fbb145fda0c935b764824b02]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250626 20:07:26.186342 24551 consensus_peers.cc:487] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 -> Peer 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699): Couldn't send request to peer 77208ca02c274a27a7ed27c2fed0ac69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7dfd28923b74a75b1f4df0aa6caab31. This is attempt 1: this message will repeat every 5th retry.
W20250626 20:07:26.240384 24667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:26.240500 24667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:26.240517 24667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:26.241680 24667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:26.241731 24667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:26.242926 24667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:42075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=41437
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.24667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:26.243073 24667 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:26.243213 24667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:26.246140 24680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:26.246238 24681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:26.246281 24667 server_base.cc:1048] running on GCE node
W20250626 20:07:26.246407 24683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:26.246587 24667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:26.246740 24667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:26.247857 24667 hybrid_clock.cc:648] HybridClock initialized: now 1750968446247845 us; error 24 us; skew 500 ppm
I20250626 20:07:26.248773 24667 webserver.cc:469] Webserver started at http://127.20.190.195:41437/ using document root <none> and password file <none>
I20250626 20:07:26.248947 24667 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:26.248996 24667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:26.249909 24667 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:26.250349 24689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.250474 24667 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.250546 24667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "672299fad8844abd851a2a853328394c"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:26.250720 24667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:26.261976 24667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:26.262171 24667 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:26.262260 24667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:26.262413 24667 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:26.262754 24696 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250626 20:07:26.263334 24667 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 20:07:26.263366 24667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:26.263399 24667 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 20:07:26.263808 24667 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 20:07:26.263835 24667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.263875 24696 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap starting.
I20250626 20:07:26.269147 24667 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:42075
I20250626 20:07:26.269213 24803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:42075 every 8 connection(s)
I20250626 20:07:26.269409 24667 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:26.272979 24804 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:26.273030 24696 log.cc:826] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:26.273061 24804 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:26.273231 24804 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:26.273626 24337 ts_manager.cc:194] Registered new tserver with Master: 672299fad8844abd851a2a853328394c (127.20.190.195:42075)
I20250626 20:07:26.273947 24337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:37115
I20250626 20:07:26.276265 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 24667
I20250626 20:07:26.279367 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:26.279780 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:26.279798 24696 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=39 ignored=0} inserts{seen=1900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.280017 24696 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap complete.
W20250626 20:07:26.280165 21243 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250626 20:07:26.280730 24696 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20250626 20:07:26.281307 24696 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:26.281451 24696 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672299fad8844abd851a2a853328394c, State: Initialized, Role: FOLLOWER
I20250626 20:07:26.281538 24696 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:26.281579 24696 raft_consensus.cc:397] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:26.281601 24696 raft_consensus.cc:491] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:26.281641 24696 raft_consensus.cc:3058] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:26.282250 24696 raft_consensus.cc:513] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:26.282325 24696 leader_election.cc:304] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672299fad8844abd851a2a853328394c; no voters:
I20250626 20:07:26.282449 24696 leader_election.cc:290] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 20:07:26.282491 24809 raft_consensus.cc:2802] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:26.282616 24696 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250626 20:07:26.282637 24804 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:26.282640 24809 raft_consensus.cc:695] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 LEADER]: Becoming Leader. State: Replica: 672299fad8844abd851a2a853328394c, State: Running, Role: LEADER
I20250626 20:07:26.282732 24809 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } }
I20250626 20:07:26.283165 24337 catalog_manager.cc:5582] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c reported cstate change: term changed from 0 to 2, leader changed from <none> to 672299fad8844abd851a2a853328394c (127.20.190.195), VOTER 672299fad8844abd851a2a853328394c (127.20.190.195) added. New cstate: current_term: 2 leader_uuid: "672299fad8844abd851a2a853328394c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:26.284303 24617 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 32, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:26.284633 24668 raft_consensus.cc:2953] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER 672299fad8844abd851a2a853328394c (127.20.190.195) added. New config: { opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } } }
W20250626 20:07:26.285113 24551 consensus_peers.cc:487] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 -> Peer 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699): Couldn't send request to peer 77208ca02c274a27a7ed27c2fed0ac69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7dfd28923b74a75b1f4df0aa6caab31. This is attempt 1: this message will repeat every 5th retry.
I20250626 20:07:26.285408 24323 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 with cas_config_opid_index 34: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20250626 20:07:26.285624 24337 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: config changed from index 34 to 35, NON_VOTER 672299fad8844abd851a2a853328394c (127.20.190.195) added. New cstate: current_term: 2 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:26.286863 24758 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 39, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } }
I20250626 20:07:26.287235 24811 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index -1 to 41, NON_VOTER fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) added. New config: { opid_index: 41 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } }
W20250626 20:07:26.287870 24551 consensus_peers.cc:487] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 -> Peer 672299fad8844abd851a2a853328394c (127.20.190.195:42075): Couldn't send request to peer 672299fad8844abd851a2a853328394c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7dfd28923b74a75b1f4df0aa6caab31. This is attempt 1: this message will repeat every 5th retry.
I20250626 20:07:26.288017 24323 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 30adbf13379941e2b1c4857ae476b139 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 20:07:26.288425 24337 catalog_manager.cc:5582] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c reported cstate change: config changed from index -1 to 41, NON_VOTER fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) added. New cstate: current_term: 2 leader_uuid: "672299fad8844abd851a2a853328394c" committed_config { opid_index: 41 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:26.288978 24758 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 39, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:26.289211 24810 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 LEADER]: Committing config change with OpId 2.42: config changed from index 41 to 42, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) added. New config: { opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } }
I20250626 20:07:26.289752 24323 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 30adbf13379941e2b1c4857ae476b139 with cas_config_opid_index 41: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 20:07:26.289988 24337 catalog_manager.cc:5582] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c reported cstate change: config changed from index 41 to 42, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) added. New cstate: current_term: 2 leader_uuid: "672299fad8844abd851a2a853328394c" committed_config { opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 20:07:26.290104 24692 consensus_peers.cc:487] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c -> Peer fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511): Couldn't send request to peer fbfca0cc6a254dab9b1efa2dcf374770. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30adbf13379941e2b1c4857ae476b139. This is attempt 1: this message will repeat every 5th retry.
W20250626 20:07:26.291843 24692 consensus_peers.cc:487] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c -> Peer 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699): Couldn't send request to peer 77208ca02c274a27a7ed27c2fed0ac69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30adbf13379941e2b1c4857ae476b139. This is attempt 1: this message will repeat every 5th retry.
I20250626 20:07:26.660285 24816 ts_tablet_manager.cc:927] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Initiating tablet copy from peer fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:26.660465 24816 tablet_copy_client.cc:323] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Beginning tablet copy session from remote peer at address 127.20.190.194:45511
I20250626 20:07:26.662846 24637 tablet_copy_service.cc:140] P fbfca0cc6a254dab9b1efa2dcf374770: Received BeginTabletCopySession request for tablet d7dfd28923b74a75b1f4df0aa6caab31 from peer 77208ca02c274a27a7ed27c2fed0ac69 ({username='slave'} at 127.20.190.193:56881)
I20250626 20:07:26.662917 24637 tablet_copy_service.cc:161] P fbfca0cc6a254dab9b1efa2dcf374770: Beginning new tablet copy session on tablet d7dfd28923b74a75b1f4df0aa6caab31 from peer 77208ca02c274a27a7ed27c2fed0ac69 at {username='slave'} at 127.20.190.193:56881: session id = 77208ca02c274a27a7ed27c2fed0ac69-d7dfd28923b74a75b1f4df0aa6caab31
I20250626 20:07:26.663295 24637 tablet_copy_source_session.cc:215] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 20:07:26.663808 24816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7dfd28923b74a75b1f4df0aa6caab31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:26.664997 24816 tablet_copy_client.cc:806] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Starting download of 0 data blocks...
I20250626 20:07:26.665108 24816 tablet_copy_client.cc:670] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Starting download of 1 WAL segments...
I20250626 20:07:26.665798 24816 tablet_copy_client.cc:538] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 20:07:26.666380 24816 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:26.671409 24816 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.671588 24816 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap complete.
I20250626 20:07:26.671679 24816 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250626 20:07:26.671777 24816 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:26.671828 24816 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: LEARNER
I20250626 20:07:26.671851 24816 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:26.672003 24816 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:26.672206 24637 tablet_copy_service.cc:342] P fbfca0cc6a254dab9b1efa2dcf374770: Request end of tablet copy session 77208ca02c274a27a7ed27c2fed0ac69-d7dfd28923b74a75b1f4df0aa6caab31 received from {username='slave'} at 127.20.190.193:56881
I20250626 20:07:26.672257 24637 tablet_copy_service.cc:434] P fbfca0cc6a254dab9b1efa2dcf374770: ending tablet copy session 77208ca02c274a27a7ed27c2fed0ac69-d7dfd28923b74a75b1f4df0aa6caab31 on tablet d7dfd28923b74a75b1f4df0aa6caab31 with peer 77208ca02c274a27a7ed27c2fed0ac69
I20250626 20:07:26.693042 24821 ts_tablet_manager.cc:927] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Initiating tablet copy from peer fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:26.693202 24821 tablet_copy_client.cc:323] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.190.194:45511
I20250626 20:07:26.693449 24637 tablet_copy_service.cc:140] P fbfca0cc6a254dab9b1efa2dcf374770: Received BeginTabletCopySession request for tablet d7dfd28923b74a75b1f4df0aa6caab31 from peer 672299fad8844abd851a2a853328394c ({username='slave'} at 127.20.190.195:41863)
I20250626 20:07:26.693493 24637 tablet_copy_service.cc:161] P fbfca0cc6a254dab9b1efa2dcf374770: Beginning new tablet copy session on tablet d7dfd28923b74a75b1f4df0aa6caab31 from peer 672299fad8844abd851a2a853328394c at {username='slave'} at 127.20.190.195:41863: session id = 672299fad8844abd851a2a853328394c-d7dfd28923b74a75b1f4df0aa6caab31
I20250626 20:07:26.693816 24637 tablet_copy_source_session.cc:215] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 20:07:26.694159 24821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7dfd28923b74a75b1f4df0aa6caab31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:26.695209 24821 tablet_copy_client.cc:806] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: tablet copy: Starting download of 0 data blocks...
I20250626 20:07:26.695324 24821 tablet_copy_client.cc:670] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: tablet copy: Starting download of 1 WAL segments...
I20250626 20:07:26.695989 24821 tablet_copy_client.cc:538] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 20:07:26.696477 24821 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Bootstrap starting.
I20250626 20:07:26.701562 24821 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.701697 24821 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Bootstrap complete.
I20250626 20:07:26.701766 24821 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20250626 20:07:26.701853 24821 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:26.701896 24821 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 672299fad8844abd851a2a853328394c, State: Initialized, Role: LEARNER
I20250626 20:07:26.701912 24821 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:26.701977 24821 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.702157 24637 tablet_copy_service.cc:342] P fbfca0cc6a254dab9b1efa2dcf374770: Request end of tablet copy session 672299fad8844abd851a2a853328394c-d7dfd28923b74a75b1f4df0aa6caab31 received from {username='slave'} at 127.20.190.195:41863
I20250626 20:07:26.702198 24637 tablet_copy_service.cc:434] P fbfca0cc6a254dab9b1efa2dcf374770: ending tablet copy session 672299fad8844abd851a2a853328394c-d7dfd28923b74a75b1f4df0aa6caab31 on tablet d7dfd28923b74a75b1f4df0aa6caab31 with peer 672299fad8844abd851a2a853328394c
I20250626 20:07:26.787979 24824 ts_tablet_manager.cc:927] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Initiating tablet copy from peer 672299fad8844abd851a2a853328394c (127.20.190.195:42075)
I20250626 20:07:26.788141 24824 tablet_copy_client.cc:323] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: tablet copy: Beginning tablet copy session from remote peer at address 127.20.190.195:42075
I20250626 20:07:26.788404 24778 tablet_copy_service.cc:140] P 672299fad8844abd851a2a853328394c: Received BeginTabletCopySession request for tablet 30adbf13379941e2b1c4857ae476b139 from peer fbfca0cc6a254dab9b1efa2dcf374770 ({username='slave'} at 127.20.190.194:48553)
I20250626 20:07:26.788468 24778 tablet_copy_service.cc:161] P 672299fad8844abd851a2a853328394c: Beginning new tablet copy session on tablet 30adbf13379941e2b1c4857ae476b139 from peer fbfca0cc6a254dab9b1efa2dcf374770 at {username='slave'} at 127.20.190.194:48553: session id = fbfca0cc6a254dab9b1efa2dcf374770-30adbf13379941e2b1c4857ae476b139
I20250626 20:07:26.788846 24778 tablet_copy_source_session.cc:215] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 20:07:26.789245 24824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30adbf13379941e2b1c4857ae476b139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:26.790383 24824 tablet_copy_client.cc:806] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: tablet copy: Starting download of 0 data blocks...
I20250626 20:07:26.790537 24824 tablet_copy_client.cc:670] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: tablet copy: Starting download of 1 WAL segments...
I20250626 20:07:26.791374 24824 tablet_copy_client.cc:538] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 20:07:26.791862 24824 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap starting.
I20250626 20:07:26.797992 24824 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=1900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.798128 24824 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap complete.
I20250626 20:07:26.798230 24824 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250626 20:07:26.798347 24824 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:26.798398 24824 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Initialized, Role: LEARNER
I20250626 20:07:26.798418 24824 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 2.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:26.798528 24824 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.798745 24778 tablet_copy_service.cc:342] P 672299fad8844abd851a2a853328394c: Request end of tablet copy session fbfca0cc6a254dab9b1efa2dcf374770-30adbf13379941e2b1c4857ae476b139 received from {username='slave'} at 127.20.190.194:48553
I20250626 20:07:26.798803 24778 tablet_copy_service.cc:434] P 672299fad8844abd851a2a853328394c: ending tablet copy session fbfca0cc6a254dab9b1efa2dcf374770-30adbf13379941e2b1c4857ae476b139 on tablet 30adbf13379941e2b1c4857ae476b139 with peer fbfca0cc6a254dab9b1efa2dcf374770
I20250626 20:07:26.816069 24816 ts_tablet_manager.cc:927] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Initiating tablet copy from peer 672299fad8844abd851a2a853328394c (127.20.190.195:42075)
I20250626 20:07:26.816198 24816 tablet_copy_client.cc:323] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Beginning tablet copy session from remote peer at address 127.20.190.195:42075
I20250626 20:07:26.818397 24778 tablet_copy_service.cc:140] P 672299fad8844abd851a2a853328394c: Received BeginTabletCopySession request for tablet 30adbf13379941e2b1c4857ae476b139 from peer 77208ca02c274a27a7ed27c2fed0ac69 ({username='slave'} at 127.20.190.193:56205)
I20250626 20:07:26.818442 24778 tablet_copy_service.cc:161] P 672299fad8844abd851a2a853328394c: Beginning new tablet copy session on tablet 30adbf13379941e2b1c4857ae476b139 from peer 77208ca02c274a27a7ed27c2fed0ac69 at {username='slave'} at 127.20.190.193:56205: session id = 77208ca02c274a27a7ed27c2fed0ac69-30adbf13379941e2b1c4857ae476b139
I20250626 20:07:26.818926 24778 tablet_copy_source_session.cc:215] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 20:07:26.819217 24816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30adbf13379941e2b1c4857ae476b139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:26.820039 24816 tablet_copy_client.cc:806] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Starting download of 0 data blocks...
I20250626 20:07:26.820128 24816 tablet_copy_client.cc:670] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Starting download of 1 WAL segments...
I20250626 20:07:26.820708 24816 tablet_copy_client.cc:538] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 20:07:26.821161 24816 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:26.827390 24816 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=1900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:26.827553 24816 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap complete.
I20250626 20:07:26.827642 24816 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20250626 20:07:26.827751 24816 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:26.827802 24816 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: LEARNER
I20250626 20:07:26.827824 24816 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 2.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: NON_VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: true } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:26.827901 24816 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:26.828115 24778 tablet_copy_service.cc:342] P 672299fad8844abd851a2a853328394c: Request end of tablet copy session 77208ca02c274a27a7ed27c2fed0ac69-30adbf13379941e2b1c4857ae476b139 received from {username='slave'} at 127.20.190.193:56205
I20250626 20:07:26.828167 24778 tablet_copy_service.cc:434] P 672299fad8844abd851a2a853328394c: ending tablet copy session 77208ca02c274a27a7ed27c2fed0ac69-30adbf13379941e2b1c4857ae476b139 on tablet 30adbf13379941e2b1c4857ae476b139 with peer 77208ca02c274a27a7ed27c2fed0ac69
I20250626 20:07:27.201463 24617 raft_consensus.cc:1215] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.41->[2.42-2.42] Dedup: 2.42->[]
I20250626 20:07:27.223608 24477 raft_consensus.cc:1215] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.41->[2.42-2.42] Dedup: 2.42->[]
I20250626 20:07:27.271286 24477 raft_consensus.cc:1215] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.34->[2.35-2.35] Dedup: 2.35->[]
I20250626 20:07:27.280903 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 77208ca02c274a27a7ed27c2fed0ac69 to finish bootstrapping
I20250626 20:07:27.286453 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fbfca0cc6a254dab9b1efa2dcf374770 to finish bootstrapping
I20250626 20:07:27.289112 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 672299fad8844abd851a2a853328394c to finish bootstrapping
I20250626 20:07:27.295855 24758 raft_consensus.cc:1215] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.34->[2.35-2.35] Dedup: 2.35->[]
I20250626 20:07:27.312629 24738 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:27.312655 24597 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:27.316946 24438 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
677637a36d1e405bb222c2c8bea54649 | 127.20.190.254:46165 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:34167 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
672299fad8844abd851a2a853328394c | 127.20.190.195:42075 | HEALTHY | <none> | 1 | 0
77208ca02c274a27a7ed27c2fed0ac69 | 127.20.190.193:37699 | HEALTHY | <none> | 1 | 0
fbfca0cc6a254dab9b1efa2dcf374770 | 127.20.190.194:45511 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.190.193 | experimental | 127.20.190.193:37699
local_ip_for_outbound_sockets | 127.20.190.194 | experimental | 127.20.190.194:45511
local_ip_for_outbound_sockets | 127.20.190.195 | experimental | 127.20.190.195:42075
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.190.193:37699
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.190.194:45511
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.190.195:42075
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:34167 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 20:07:27.325153 21243 log_verifier.cc:126] Checking tablet 30adbf13379941e2b1c4857ae476b139
I20250626 20:07:27.344050 21243 log_verifier.cc:177] Verified matching terms for 42 ops in tablet 30adbf13379941e2b1c4857ae476b139
I20250626 20:07:27.344108 21243 log_verifier.cc:126] Checking tablet 43cae63b9def4f79b9e1a3d28751e7b4
I20250626 20:07:27.349471 21243 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 43cae63b9def4f79b9e1a3d28751e7b4
I20250626 20:07:27.349510 21243 log_verifier.cc:126] Checking tablet d7dfd28923b74a75b1f4df0aa6caab31
I20250626 20:07:27.367542 21243 log_verifier.cc:177] Verified matching terms for 35 ops in tablet d7dfd28923b74a75b1f4df0aa6caab31
I20250626 20:07:27.367640 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 24306
I20250626 20:07:27.372875 21243 minidump.cc:252] Setting minidump size limit to 20M
I20250626 20:07:27.373091 21243 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:27.373205 21243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:27.374965 24865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:27.374972 24862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:27.375161 21243 server_base.cc:1048] running on GCE node
W20250626 20:07:27.375201 24863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:27.375375 21243 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 20:07:27.375412 21243 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 20:07:27.375442 21243 hybrid_clock.cc:648] HybridClock initialized: now 1750968447375441 us; error 0 us; skew 500 ppm
I20250626 20:07:27.376135 21243 webserver.cc:469] Webserver started at http://0.0.0.0:43407/ using document root <none> and password file <none>
I20250626 20:07:27.376296 21243 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:27.376334 21243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:27.377051 21243 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:27.377485 24870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:27.377611 21243 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:27.377672 21243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "677637a36d1e405bb222c2c8bea54649"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:27.377838 21243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:27.389972 21243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:27.390117 21243 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:27.390199 21243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:27.391010 21243 sys_catalog.cc:263] Verifying existing consensus state
W20250626 20:07:27.391295 21243 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.20.190.254:46165 exists but no master address supplied!
I20250626 20:07:27.391757 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap starting.
I20250626 20:07:27.398208 21243 log.cc:826] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:27.400144 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=13 ignored=0} mutations{seen=17 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:27.400350 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap complete.
I20250626 20:07:27.401664 21243 raft_consensus.cc:357] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.401755 21243 raft_consensus.cc:738] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Initialized, Role: FOLLOWER
I20250626 20:07:27.401834 21243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.401909 21243 raft_consensus.cc:397] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:27.401952 21243 raft_consensus.cc:491] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:27.401993 21243 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 3 FOLLOWER]: Advancing to term 4
I20250626 20:07:27.402735 21243 raft_consensus.cc:513] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.402827 21243 leader_election.cc:304] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677637a36d1e405bb222c2c8bea54649; no voters:
I20250626 20:07:27.402945 21243 leader_election.cc:290] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250626 20:07:27.402984 24877 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 20:07:27.403128 24877 raft_consensus.cc:695] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 4 LEADER]: Becoming Leader. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Running, Role: LEADER
I20250626 20:07:27.403204 24877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.403425 24878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:27.403470 24879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677637a36d1e405bb222c2c8bea54649. Latest consensus state: current_term: 4 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:27.403584 24879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:27.403563 24878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:27.404834 21243 tablet_replica.cc:331] stopping tablet replica
I20250626 20:07:27.404902 21243 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 4 LEADER]: Raft consensus shutting down.
I20250626 20:07:27.404945 21243 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 20:07:27.405212 21243 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 20:07:27.405323 21243 acceptor_pool.cc:194] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 20:07:27.416621 21243 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250626 20:07:27.643154 24831 raft_consensus.cc:1062] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: attempting to promote NON_VOTER fbfca0cc6a254dab9b1efa2dcf374770 to VOTER
I20250626 20:07:27.643326 24831 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 2.42, Last appended by leader: 39, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } }
I20250626 20:07:27.643797 24477 raft_consensus.cc:1273] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Refusing update from remote peer 672299fad8844abd851a2a853328394c: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20250626 20:07:27.643810 24617 raft_consensus.cc:1273] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 LEARNER]: Refusing update from remote peer 672299fad8844abd851a2a853328394c: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20250626 20:07:27.643998 24809 consensus_queue.cc:1035] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250626 20:07:27.644083 24830 consensus_queue.cc:1035] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250626 20:07:27.645058 24809 raft_consensus.cc:1025] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 LEADER]: attempt to promote peer 77208ca02c274a27a7ed27c2fed0ac69: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250626 20:07:27.645162 24831 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 LEADER]: Committing config change with OpId 2.43: config changed from index 42 to 43, fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } }
I20250626 20:07:27.646234 24617 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Committing config change with OpId 2.43: config changed from index 42 to 43, fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } }
W20250626 20:07:27.646936 24804 heartbeater.cc:646] Failed to heartbeat to 127.20.190.254:46165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.190.254:46165: connect: Connection refused (error 111)
I20250626 20:07:27.647332 24477 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Committing config change with OpId 2.43: config changed from index 42 to 43, fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } }
I20250626 20:07:27.648298 24809 raft_consensus.cc:1062] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: attempting to promote NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 to VOTER
W20250626 20:07:27.648303 24663 heartbeater.cc:646] Failed to heartbeat to 127.20.190.254:46165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.190.254:46165: connect: Connection refused (error 111)
I20250626 20:07:27.648432 24809 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 2.43, Last appended by leader: 39, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
W20250626 20:07:27.648583 24523 heartbeater.cc:646] Failed to heartbeat to 127.20.190.254:46165 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.190.254:46165: connect: Connection refused (error 111)
I20250626 20:07:27.648720 24477 raft_consensus.cc:1273] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Refusing update from remote peer 672299fad8844abd851a2a853328394c: Log matching property violated. Preceding OpId in replica: term: 2 index: 43. Preceding OpId from leader: term: 2 index: 44. (index mismatch)
I20250626 20:07:27.648833 24617 raft_consensus.cc:1273] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Refusing update from remote peer 672299fad8844abd851a2a853328394c: Log matching property violated. Preceding OpId in replica: term: 2 index: 43. Preceding OpId from leader: term: 2 index: 44. (index mismatch)
I20250626 20:07:27.648980 24831 consensus_queue.cc:1035] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250626 20:07:27.649055 24830 consensus_queue.cc:1035] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250626 20:07:27.649566 24831 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 LEADER]: Committing config change with OpId 2.44: config changed from index 43 to 44, 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } } }
I20250626 20:07:27.649672 24617 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Committing config change with OpId 2.44: config changed from index 43 to 44, 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } } }
I20250626 20:07:27.650741 24477 raft_consensus.cc:2953] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Committing config change with OpId 2.44: config changed from index 43 to 44, 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } } }
I20250626 20:07:27.692903 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 24376
I20250626 20:07:27.697284 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 24528
I20250626 20:07:27.701114 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 24667
I20250626 20:07:27.705154 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:46165
--webserver_interface=127.20.190.254
--webserver_port=42993
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:46165 with env {}
W20250626 20:07:27.770376 24933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:27.770504 24933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:27.770534 24933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:27.771723 24933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:27.771759 24933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:27.771771 24933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:27.771781 24933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:27.772964 24933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:46165
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=42993
--never_fsync=true
--heap_profile_path=/tmp/kudu.24933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:27.773123 24933 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:27.773295 24933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:27.776225 24941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:27.776232 24938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:27.776280 24939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:27.776435 24933 server_base.cc:1048] running on GCE node
I20250626 20:07:27.776603 24933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:27.776772 24933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:27.777889 24933 hybrid_clock.cc:648] HybridClock initialized: now 1750968447777875 us; error 31 us; skew 500 ppm
I20250626 20:07:27.778859 24933 webserver.cc:469] Webserver started at http://127.20.190.254:42993/ using document root <none> and password file <none>
I20250626 20:07:27.779023 24933 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:27.779060 24933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:27.779959 24933 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:27.780400 24947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:27.780514 24933 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:27.780571 24933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "677637a36d1e405bb222c2c8bea54649"
format_stamp: "Formatted at 2025-06-26 20:07:24 on dist-test-slave-l47t"
I20250626 20:07:27.780754 24933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:27.788435 24933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:27.788642 24933 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:27.788739 24933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:27.791972 24933 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:46165
I20250626 20:07:27.792004 24999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:46165 every 8 connection(s)
I20250626 20:07:27.792243 24933 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:27.793061 25000 sys_catalog.cc:263] Verifying existing consensus state
I20250626 20:07:27.793501 25000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap starting.
I20250626 20:07:27.794667 25000 log.cc:826] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:27.796931 25000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=15 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:27.797173 25000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Bootstrap complete.
I20250626 20:07:27.798161 25000 raft_consensus.cc:357] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.798312 25000 raft_consensus.cc:738] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Initialized, Role: FOLLOWER
I20250626 20:07:27.798389 25000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.798426 25000 raft_consensus.cc:397] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:27.798445 25000 raft_consensus.cc:491] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:27.798466 25000 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 5 FOLLOWER]: Advancing to term 6
I20250626 20:07:27.798844 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 24933
I20250626 20:07:27.799028 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:37699
--local_ip_for_outbound_sockets=127.20.190.193
--tserver_master_addrs=127.20.190.254:46165
--webserver_port=37031
--webserver_interface=127.20.190.193
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:27.799233 25000 raft_consensus.cc:513] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.799312 25000 leader_election.cc:304] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677637a36d1e405bb222c2c8bea54649; no voters:
I20250626 20:07:27.799428 25000 leader_election.cc:290] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250626 20:07:27.799474 25003 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 20:07:27.799597 25003 raft_consensus.cc:695] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [term 6 LEADER]: Becoming Leader. State: Replica: 677637a36d1e405bb222c2c8bea54649, State: Running, Role: LEADER
I20250626 20:07:27.799623 25000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:27.799712 25003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } }
I20250626 20:07:27.800009 25005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677637a36d1e405bb222c2c8bea54649. Latest consensus state: current_term: 6 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:27.800071 25005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:27.800333 25004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "677637a36d1e405bb222c2c8bea54649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677637a36d1e405bb222c2c8bea54649" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 46165 } } }
I20250626 20:07:27.800401 25004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:27.800320 25007 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:27.800877 25007 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=4e861185dbcd47f293f7a44305abba55]
I20250626 20:07:27.801172 25007 catalog_manager.cc:671] Loaded metadata for table TestTable [id=532f9d378ee649aaad94df77a867f917]
I20250626 20:07:27.801224 25007 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=ecc6bf3164db4166b6f547db186842fa]
I20250626 20:07:27.801384 25007 tablet_loader.cc:96] loaded metadata for tablet 30adbf13379941e2b1c4857ae476b139 (table TestTable1 [id=4e861185dbcd47f293f7a44305abba55])
I20250626 20:07:27.801433 25007 tablet_loader.cc:96] loaded metadata for tablet 43cae63b9def4f79b9e1a3d28751e7b4 (table TestTable2 [id=ecc6bf3164db4166b6f547db186842fa])
I20250626 20:07:27.801467 25007 tablet_loader.cc:96] loaded metadata for tablet d7dfd28923b74a75b1f4df0aa6caab31 (table TestTable [id=532f9d378ee649aaad94df77a867f917])
I20250626 20:07:27.801498 25007 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:27.801587 25007 catalog_manager.cc:1261] Loaded cluster ID: f44250e2f8bb460dae5e937c802164a9
I20250626 20:07:27.801610 25007 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:27.802374 25007 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:27.802536 25007 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 677637a36d1e405bb222c2c8bea54649: Loaded TSK: 0
I20250626 20:07:27.802732 25007 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 20:07:27.870127 25010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:27.870260 25010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:27.870275 25010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:27.871457 25010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:27.871501 25010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:27.872668 25010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:37699
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=37031
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.25010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:27.872857 25010 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:27.873036 25010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:27.876192 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:27.876199 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:27.876176 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:27.876376 25010 server_base.cc:1048] running on GCE node
I20250626 20:07:27.876557 25010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:27.876775 25010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:27.877931 25010 hybrid_clock.cc:648] HybridClock initialized: now 1750968447877913 us; error 35 us; skew 500 ppm
I20250626 20:07:27.879067 25010 webserver.cc:469] Webserver started at http://127.20.190.193:37031/ using document root <none> and password file <none>
I20250626 20:07:27.879259 25010 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:27.879312 25010 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:27.880556 25010 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:27.881247 25035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:27.881387 25010 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:27.881467 25010 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "77208ca02c274a27a7ed27c2fed0ac69"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:27.881721 25010 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:27.910666 25010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:27.910984 25010 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:27.911118 25010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:27.911373 25010 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:27.911847 25042 ts_tablet_manager.cc:536] Loading tablet metadata (0/3 complete)
I20250626 20:07:27.913029 25010 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250626 20:07:27.913091 25010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:27.913125 25010 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250626 20:07:27.913908 25042 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:27.914415 25010 ts_tablet_manager.cc:610] Registered 3 tablets
I20250626 20:07:27.914474 25010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:27.921077 25042 log.cc:826] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:27.921348 25010 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:37699
I20250626 20:07:27.921713 25010 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:27.926776 25150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:37699 every 8 connection(s)
I20250626 20:07:27.927609 25042 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:27.927901 25042 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap complete.
I20250626 20:07:27.928998 25042 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20250626 20:07:27.929144 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25010
I20250626 20:07:27.929407 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:45511
--local_ip_for_outbound_sockets=127.20.190.194
--tserver_master_addrs=127.20.190.254:46165
--webserver_port=45971
--webserver_interface=127.20.190.194
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:27.929967 25042 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:27.930223 25042 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: LEARNER
I20250626 20:07:27.930294 25042 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:27.930533 25042 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20250626 20:07:27.930639 25042 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:27.932050 25151 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:27.932144 25151 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:27.932318 25151 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:27.933036 24964 ts_manager.cc:194] Registered new tserver with Master: 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699)
I20250626 20:07:27.933238 24964 catalog_manager.cc:5582] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 reported cstate change: config changed from index 42 to 44, 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) changed from NON_VOTER to VOTER, fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "672299fad8844abd851a2a853328394c" committed_config { opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } } }
I20250626 20:07:27.933406 24964 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 reported cstate change: config changed from index -1 to 35, term changed from 0 to 2, NON_VOTER 672299fad8844abd851a2a853328394c (127.20.190.195) added, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) added, VOTER fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194) added. New cstate: current_term: 2 committed_config { opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } } }
I20250626 20:07:27.937496 24964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:56267
I20250626 20:07:27.938359 25151 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:27.944453 25042 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:27.944697 25042 tablet_bootstrap.cc:492] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap complete.
I20250626 20:07:27.945752 25042 ts_tablet_manager.cc:1397] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20250626 20:07:27.945943 25042 raft_consensus.cc:357] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:27.946012 25042 raft_consensus.cc:738] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: FOLLOWER
I20250626 20:07:27.946087 25042 consensus_queue.cc:260] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:27.946125 25042 raft_consensus.cc:397] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:27.946163 25042 raft_consensus.cc:491] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:27.946205 25042 raft_consensus.cc:3058] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:27.947355 25042 raft_consensus.cc:513] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:27.947460 25042 leader_election.cc:304] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77208ca02c274a27a7ed27c2fed0ac69; no voters:
I20250626 20:07:27.947530 25042 leader_election.cc:290] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 20:07:27.947564 25155 raft_consensus.cc:2802] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 20:07:27.947666 25042 ts_tablet_manager.cc:1428] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
I20250626 20:07:27.947693 25155 raft_consensus.cc:695] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 LEADER]: Becoming Leader. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Running, Role: LEADER
I20250626 20:07:27.947724 25042 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap starting.
I20250626 20:07:27.947767 25155 consensus_queue.cc:237] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } }
I20250626 20:07:27.948261 24964 catalog_manager.cc:5582] T 43cae63b9def4f79b9e1a3d28751e7b4 P 77208ca02c274a27a7ed27c2fed0ac69 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "77208ca02c274a27a7ed27c2fed0ac69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:27.960461 25042 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=1900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:27.960691 25042 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Bootstrap complete.
I20250626 20:07:27.961606 25042 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20250626 20:07:27.961704 25042 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:27.961759 25042 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Initialized, Role: FOLLOWER
I20250626 20:07:27.961794 25042 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:27.961875 25042 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20250626 20:07:28.011476 25156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:28.011610 25156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:28.011627 25156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:28.012836 25156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:28.012882 25156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:28.014111 25156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:45511
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=45971
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.25156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:28.014269 25156 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:28.014449 25156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:28.017540 25168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:28.017593 25171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:28.017683 25156 server_base.cc:1048] running on GCE node
W20250626 20:07:28.017712 25169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:28.017843 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:28.017984 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:28.019093 25156 hybrid_clock.cc:648] HybridClock initialized: now 1750968448019082 us; error 22 us; skew 500 ppm
I20250626 20:07:28.020041 25156 webserver.cc:469] Webserver started at http://127.20.190.194:45971/ using document root <none> and password file <none>
I20250626 20:07:28.020201 25156 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:28.020241 25156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:28.021297 25156 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:28.021855 25177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:28.021983 25156 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:28.022045 25156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fbfca0cc6a254dab9b1efa2dcf374770"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:28.022224 25156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:28.069156 25156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:28.069463 25156 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:28.069587 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:28.069810 25156 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:28.070176 25184 ts_tablet_manager.cc:536] Loading tablet metadata (0/2 complete)
I20250626 20:07:28.071112 25156 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 20:07:28.071149 25156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:28.071180 25156 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 20:07:28.071704 25184 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap starting.
I20250626 20:07:28.071841 25156 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 20:07:28.071870 25156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:28.077908 25156 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:45511
I20250626 20:07:28.078156 25184 log.cc:826] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:28.078195 25156 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:28.078367 25291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:45511 every 8 connection(s)
I20250626 20:07:28.082572 25293 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:28.082672 25293 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:28.082829 25293 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:28.083227 24964 ts_manager.cc:194] Registered new tserver with Master: fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:28.083549 24964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:34987
I20250626 20:07:28.085177 25184 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:28.085423 25184 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap complete.
I20250626 20:07:28.086277 25184 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20250626 20:07:28.086968 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25156
I20250626 20:07:28.087047 25184 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.087227 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:42075
--local_ip_for_outbound_sockets=127.20.190.195
--tserver_master_addrs=127.20.190.254:46165
--webserver_port=41437
--webserver_interface=127.20.190.195
--builtin_ntp_servers=127.20.190.212:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 20:07:28.087258 25184 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Initialized, Role: FOLLOWER
I20250626 20:07:28.087337 25184 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.087385 25184 raft_consensus.cc:397] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:28.087416 25184 raft_consensus.cc:491] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:28.087468 25184 raft_consensus.cc:3058] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:28.088239 25184 raft_consensus.cc:513] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.088335 25184 leader_election.cc:304] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbfca0cc6a254dab9b1efa2dcf374770; no voters:
I20250626 20:07:28.088488 25184 leader_election.cc:290] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 20:07:28.088634 25298 raft_consensus.cc:2802] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 20:07:28.088649 25184 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent starting tablet: real 0.002s user 0.006s sys 0.001s
I20250626 20:07:28.088734 25293 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:28.088739 25184 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap starting.
I20250626 20:07:28.088888 25298 raft_consensus.cc:695] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 LEADER]: Becoming Leader. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Running, Role: LEADER
I20250626 20:07:28.089107 25298 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.089737 24964 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: term changed from 2 to 3, leader changed from <none> to fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194). New cstate: current_term: 3 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 20:07:28.090579 25180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.190.195:42075: connect: Connection refused (error 111)
W20250626 20:07:28.094913 25180 consensus_peers.cc:487] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 -> Peer 672299fad8844abd851a2a853328394c (127.20.190.195:42075): Couldn't send request to peer 672299fad8844abd851a2a853328394c. Status: Network error: Client connection negotiation failed: client connection to 127.20.190.195:42075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 20:07:28.098191 25104 raft_consensus.cc:3058] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 LEARNER]: Advancing to term 3
I20250626 20:07:28.098996 25104 raft_consensus.cc:1273] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 LEARNER]: Refusing update from remote peer fbfca0cc6a254dab9b1efa2dcf374770: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20250626 20:07:28.099226 25246 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.36, Last appended by leader: 35, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.099571 25297 raft_consensus.cc:2953] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 LEADER]: Committing config change with OpId 3.37: config changed from index 35 to 37, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) evicted. New config: { opid_index: 37 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } } }
W20250626 20:07:28.100016 25180 consensus_peers.cc:487] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 -> Peer 672299fad8844abd851a2a853328394c (127.20.190.195:42075): Couldn't send request to peer 672299fad8844abd851a2a853328394c. Status: Network error: Client connection negotiation failed: client connection to 127.20.190.195:42075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 20:07:28.100584 24950 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 20:07:28.100775 24964 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: config changed from index 35 to 37, NON_VOTER 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193) evicted. New cstate: current_term: 3 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: 37 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 20:07:28.101733 25246 consensus_queue.cc:237] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 3.37, Last appended by leader: 35, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } }
I20250626 20:07:28.101928 25302 raft_consensus.cc:2953] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 LEADER]: Committing config change with OpId 3.38: config changed from index 37 to 38, NON_VOTER 672299fad8844abd851a2a853328394c (127.20.190.195) evicted. New config: { opid_index: 38 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } }
I20250626 20:07:28.102909 24950 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 20:07:28.103132 24964 catalog_manager.cc:5582] T d7dfd28923b74a75b1f4df0aa6caab31 P fbfca0cc6a254dab9b1efa2dcf374770 reported cstate change: config changed from index 37 to 38, NON_VOTER 672299fad8844abd851a2a853328394c (127.20.190.195) evicted. New cstate: current_term: 3 leader_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" committed_config { opid_index: 38 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } health_report { overall_health: HEALTHY } } }
W20250626 20:07:28.103749 24964 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 on TS 672299fad8844abd851a2a853328394c: Not found: failed to reset TS proxy: Could not find TS for UUID 672299fad8844abd851a2a853328394c
I20250626 20:07:28.104471 25084 tablet_service.cc:1515] Processing DeleteTablet for tablet d7dfd28923b74a75b1f4df0aa6caab31 with delete_type TABLET_DATA_TOMBSTONED (TS 77208ca02c274a27a7ed27c2fed0ac69 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:51548
I20250626 20:07:28.104856 25309 tablet_replica.cc:331] stopping tablet replica
I20250626 20:07:28.104938 25309 raft_consensus.cc:2241] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 LEARNER]: Raft consensus shutting down.
I20250626 20:07:28.104998 25309 raft_consensus.cc:2270] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 LEARNER]: Raft consensus is shut down!
I20250626 20:07:28.105239 25309 ts_tablet_manager.cc:1905] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 20:07:28.105605 25184 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=1900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:28.105851 25184 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Bootstrap complete.
I20250626 20:07:28.106384 25309 ts_tablet_manager.cc:1918] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.35
I20250626 20:07:28.106451 25309 log.cc:1199] T d7dfd28923b74a75b1f4df0aa6caab31 P 77208ca02c274a27a7ed27c2fed0ac69: Deleting WAL directory at /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7dfd28923b74a75b1f4df0aa6caab31
I20250626 20:07:28.106926 24950 catalog_manager.cc:4928] TS 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193:37699): tablet d7dfd28923b74a75b1f4df0aa6caab31 (table TestTable [id=532f9d378ee649aaad94df77a867f917]) successfully deleted
I20250626 20:07:28.106931 25184 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
I20250626 20:07:28.107045 25184 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:28.107120 25184 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfca0cc6a254dab9b1efa2dcf374770, State: Initialized, Role: FOLLOWER
I20250626 20:07:28.107182 25184 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:28.107288 25184 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20250626 20:07:28.156080 24949 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d7dfd28923b74a75b1f4df0aa6caab31 on TS 672299fad8844abd851a2a853328394c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 672299fad8844abd851a2a853328394c
W20250626 20:07:28.164935 25296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:28.165064 25296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:28.165082 25296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:28.166322 25296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:28.166365 25296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:28.167672 25296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:42075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=41437
--tserver_master_addrs=127.20.190.254:46165
--never_fsync=true
--heap_profile_path=/tmp/kudu.25296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:28.167820 25296 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:28.167958 25296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:28.170753 25315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:28.170794 25317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:28.170745 25314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:28.170944 25296 server_base.cc:1048] running on GCE node
I20250626 20:07:28.171118 25296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:28.171288 25296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:28.172406 25296 hybrid_clock.cc:648] HybridClock initialized: now 1750968448172393 us; error 24 us; skew 500 ppm
I20250626 20:07:28.173408 25296 webserver.cc:469] Webserver started at http://127.20.190.195:41437/ using document root <none> and password file <none>
I20250626 20:07:28.173573 25296 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:28.173612 25296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:28.174564 25296 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:28.175091 25323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:28.175227 25296 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:28.175288 25296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "672299fad8844abd851a2a853328394c"
format_stamp: "Formatted at 2025-06-26 20:07:25 on dist-test-slave-l47t"
I20250626 20:07:28.175486 25296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:28.183765 25296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:28.183964 25296 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:28.184064 25296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:28.184221 25296 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:28.184593 25330 ts_tablet_manager.cc:536] Loading tablet metadata (0/2 complete)
I20250626 20:07:28.185506 25296 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 20:07:28.185540 25296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:28.185560 25296 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 20:07:28.186136 25330 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Bootstrap starting.
I20250626 20:07:28.186283 25296 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 20:07:28.186316 25296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:28.191458 25296 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:42075
I20250626 20:07:28.191538 25437 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:42075 every 8 connection(s)
I20250626 20:07:28.191735 25296 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:28.192893 25330 log.cc:826] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:28.195677 25438 heartbeater.cc:344] Connected to a master server at 127.20.190.254:46165
I20250626 20:07:28.195773 25438 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:28.195915 25438 heartbeater.cc:507] Master 127.20.190.254:46165 requested a full tablet report, sending...
I20250626 20:07:28.196367 24964 ts_manager.cc:194] Registered new tserver with Master: 672299fad8844abd851a2a853328394c (127.20.190.195:42075)
I20250626 20:07:28.196885 24964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:39301
I20250626 20:07:28.199216 25330 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:28.199409 25330 tablet_bootstrap.cc:492] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Bootstrap complete.
I20250626 20:07:28.200073 25330 ts_tablet_manager.cc:1397] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250626 20:07:28.200677 25330 raft_consensus.cc:357] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.200839 25330 raft_consensus.cc:738] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 672299fad8844abd851a2a853328394c, State: Initialized, Role: LEARNER
I20250626 20:07:28.200893 25330 consensus_queue.cc:260] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: true peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: NON_VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: true } } peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: NON_VOTER last_known_addr { host: "127.20.190.195" port: 42075 } attrs { promote: true } }
I20250626 20:07:28.201043 25330 ts_tablet_manager.cc:1428] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:28.201076 25438 heartbeater.cc:499] Master 127.20.190.254:46165 was elected leader, sending a full tablet report...
I20250626 20:07:28.201105 25330 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap starting.
I20250626 20:07:28.201681 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25296
I20250626 20:07:28.205030 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:28.205545 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250626 20:07:28.206504 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250626 20:07:28.207384 25372 tablet_service.cc:1515] Processing DeleteTablet for tablet d7dfd28923b74a75b1f4df0aa6caab31 with delete_type TABLET_DATA_TOMBSTONED (TS 672299fad8844abd851a2a853328394c not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:60514
I20250626 20:07:28.207664 25446 tablet_replica.cc:331] stopping tablet replica
I20250626 20:07:28.207729 25446 raft_consensus.cc:2241] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Raft consensus shutting down.
I20250626 20:07:28.207783 25446 raft_consensus.cc:2270] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c [term 2 LEARNER]: Raft consensus is shut down!
I20250626 20:07:28.208003 25446 ts_tablet_manager.cc:1905] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 20:07:28.209225 25446 ts_tablet_manager.cc:1918] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.35
I20250626 20:07:28.209270 25446 log.cc:1199] T d7dfd28923b74a75b1f4df0aa6caab31 P 672299fad8844abd851a2a853328394c: Deleting WAL directory at /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/wals/d7dfd28923b74a75b1f4df0aa6caab31
I20250626 20:07:28.209590 24950 catalog_manager.cc:4928] TS 672299fad8844abd851a2a853328394c (127.20.190.195:42075): tablet d7dfd28923b74a75b1f4df0aa6caab31 (table TestTable [id=532f9d378ee649aaad94df77a867f917]) successfully deleted
I20250626 20:07:28.211522 25330 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=1900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:28.211712 25330 tablet_bootstrap.cc:492] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Bootstrap complete.
I20250626 20:07:28.212375 25330 ts_tablet_manager.cc:1397] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20250626 20:07:28.212493 25330 raft_consensus.cc:357] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:28.212536 25330 raft_consensus.cc:738] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672299fad8844abd851a2a853328394c, State: Initialized, Role: FOLLOWER
I20250626 20:07:28.212600 25330 consensus_queue.cc:260] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:28.212682 25330 ts_tablet_manager.cc:1428] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c: Time spent starting tablet: real 0.000s user 0.001s sys 0.001s
I20250626 20:07:29.111775 25447 raft_consensus.cc:491] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:29.111845 25447 raft_consensus.cc:513] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:29.112056 25447 leader_election.cc:290] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 672299fad8844abd851a2a853328394c (127.20.190.195:42075), fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:29.114362 25246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30adbf13379941e2b1c4857ae476b139" candidate_uuid: "77208ca02c274a27a7ed27c2fed0ac69" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" is_pre_election: true
I20250626 20:07:29.114408 25392 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30adbf13379941e2b1c4857ae476b139" candidate_uuid: "77208ca02c274a27a7ed27c2fed0ac69" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "672299fad8844abd851a2a853328394c" is_pre_election: true
I20250626 20:07:29.114463 25246 raft_consensus.cc:2466] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77208ca02c274a27a7ed27c2fed0ac69 in term 2.
I20250626 20:07:29.114552 25392 raft_consensus.cc:2466] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77208ca02c274a27a7ed27c2fed0ac69 in term 2.
I20250626 20:07:29.114637 25038 leader_election.cc:304] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77208ca02c274a27a7ed27c2fed0ac69, fbfca0cc6a254dab9b1efa2dcf374770; no voters:
I20250626 20:07:29.114741 25447 raft_consensus.cc:2802] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 20:07:29.114789 25447 raft_consensus.cc:491] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:29.114804 25447 raft_consensus.cc:3058] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:29.115336 25447 raft_consensus.cc:513] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:29.115435 25447 leader_election.cc:290] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 3 election: Requested vote from peers 672299fad8844abd851a2a853328394c (127.20.190.195:42075), fbfca0cc6a254dab9b1efa2dcf374770 (127.20.190.194:45511)
I20250626 20:07:29.115600 25246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30adbf13379941e2b1c4857ae476b139" candidate_uuid: "77208ca02c274a27a7ed27c2fed0ac69" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "fbfca0cc6a254dab9b1efa2dcf374770"
I20250626 20:07:29.115603 25392 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30adbf13379941e2b1c4857ae476b139" candidate_uuid: "77208ca02c274a27a7ed27c2fed0ac69" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "672299fad8844abd851a2a853328394c"
I20250626 20:07:29.115653 25246 raft_consensus.cc:3058] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:29.115660 25392 raft_consensus.cc:3058] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 2 FOLLOWER]: Advancing to term 3
I20250626 20:07:29.116153 25246 raft_consensus.cc:2466] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77208ca02c274a27a7ed27c2fed0ac69 in term 3.
I20250626 20:07:29.116321 25392 raft_consensus.cc:2466] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77208ca02c274a27a7ed27c2fed0ac69 in term 3.
I20250626 20:07:29.116328 25038 leader_election.cc:304] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77208ca02c274a27a7ed27c2fed0ac69, fbfca0cc6a254dab9b1efa2dcf374770; no voters:
I20250626 20:07:29.116402 25447 raft_consensus.cc:2802] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 20:07:29.116506 25447 raft_consensus.cc:695] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [term 3 LEADER]: Becoming Leader. State: Replica: 77208ca02c274a27a7ed27c2fed0ac69, State: Running, Role: LEADER
I20250626 20:07:29.116559 25447 consensus_queue.cc:237] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } }
I20250626 20:07:29.117089 24964 catalog_manager.cc:5582] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 reported cstate change: term changed from 2 to 3, leader changed from 672299fad8844abd851a2a853328394c (127.20.190.195) to 77208ca02c274a27a7ed27c2fed0ac69 (127.20.190.193). New cstate: current_term: 3 leader_uuid: "77208ca02c274a27a7ed27c2fed0ac69" committed_config { opid_index: 44 OBSOLETE_local: true peers { permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77208ca02c274a27a7ed27c2fed0ac69" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 37699 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 20:07:29.207298 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250626 20:07:29.550475 25392 raft_consensus.cc:1273] T 30adbf13379941e2b1c4857ae476b139 P 672299fad8844abd851a2a853328394c [term 3 FOLLOWER]: Refusing update from remote peer 77208ca02c274a27a7ed27c2fed0ac69: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 3 index: 45. (index mismatch)
I20250626 20:07:29.550870 25447 consensus_queue.cc:1035] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672299fad8844abd851a2a853328394c" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 42075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250626 20:07:29.552430 25246 raft_consensus.cc:1273] T 30adbf13379941e2b1c4857ae476b139 P fbfca0cc6a254dab9b1efa2dcf374770 [term 3 FOLLOWER]: Refusing update from remote peer 77208ca02c274a27a7ed27c2fed0ac69: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 3 index: 45. (index mismatch)
I20250626 20:07:29.552659 25461 consensus_queue.cc:1035] T 30adbf13379941e2b1c4857ae476b139 P 77208ca02c274a27a7ed27c2fed0ac69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfca0cc6a254dab9b1efa2dcf374770" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 45511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
W20250626 20:07:30.208127 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:31.208979 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:32.209813 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:33.210572 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:34.211329 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:35.212091 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:36.212924 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:37.213730 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:38.214581 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:39.215538 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:40.216387 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:41.217310 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:42.218055 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:43.218858 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:44.219581 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:45.220395 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:46.221216 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 20:07:47.221987 21243 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d7dfd28923b74a75b1f4df0aa6caab31: tablet_id: "d7dfd28923b74a75b1f4df0aa6caab31" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250626 20:07:48.222596 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25010
I20250626 20:07:48.226961 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25156
I20250626 20:07:48.230852 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25296
I20250626 20:07:48.234606 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 24933
2025-06-26T20:07:48Z chronyd exiting
I20250626 20:07:48.249570 21243 test_util.cc:183] -----------------------------------------------
I20250626 20:07:48.249622 21243 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750968436877382-21243-0
[ FAILED ] AdminCliTest.TestRebuildTables (23328 ms)
[----------] 5 tests from AdminCliTest (31368 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250626 20:07:48.250008 21243 test_util.cc:276] Using random seed: 495984022
I20250626 20:07:48.250435 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:48.250464 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:48.250473 21243 ts_itest-base.cc:117] 5 tablet servers
I20250626 20:07:48.250483 21243 ts_itest-base.cc:118] 3 replicas per TS
I20250626 20:07:48.250505 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:48Z Disabled control of system clock
I20250626 20:07:48.256763 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:41395
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:41395
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 20:07:48.324589 25477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:48.324714 25477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:48.324729 25477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:48.325954 25477 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 20:07:48.325994 25477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:48.326005 25477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:48.326017 25477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:48.326042 25477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:48.327394 25477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:41395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:48.327589 25477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:48.327798 25477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:48.331043 25485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.331094 25482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.331178 25483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.331231 25477 server_base.cc:1048] running on GCE node
I20250626 20:07:48.331409 25477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:48.331566 25477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:48.332679 25477 hybrid_clock.cc:648] HybridClock initialized: now 1750968468332668 us; error 25 us; skew 500 ppm
I20250626 20:07:48.333761 25477 webserver.cc:469] Webserver started at http://127.20.190.254:46329/ using document root <none> and password file <none>
I20250626 20:07:48.333941 25477 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:48.333976 25477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:48.334072 25477 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:48.334780 25477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f4f4c238ff044376b7a302ac29e1e7e8"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.335044 25477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f4f4c238ff044376b7a302ac29e1e7e8"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.335976 25477 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:48.336468 25491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.336592 25477 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:48.336654 25477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f4f4c238ff044376b7a302ac29e1e7e8"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.336705 25477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:48.350019 25477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:48.350229 25477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:48.350327 25477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:48.353372 25477 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:41395
I20250626 20:07:48.353415 25543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:41395 every 8 connection(s)
I20250626 20:07:48.353638 25477 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:48.354050 25544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:48.355860 25544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8: Bootstrap starting.
I20250626 20:07:48.356314 25544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:48.356518 25544 log.cc:826] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:48.356989 25544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8: No bootstrap required, opened a new log
I20250626 20:07:48.358009 25544 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41395 } }
I20250626 20:07:48.358114 25544 raft_consensus.cc:383] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:48.358134 25544 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f4c238ff044376b7a302ac29e1e7e8, State: Initialized, Role: FOLLOWER
I20250626 20:07:48.358227 25544 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41395 } }
I20250626 20:07:48.358294 25544 raft_consensus.cc:397] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:48.358320 25544 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:48.358366 25544 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:48.358850 25544 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41395 } }
I20250626 20:07:48.358928 25544 leader_election.cc:304] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f4c238ff044376b7a302ac29e1e7e8; no voters:
I20250626 20:07:48.359043 25544 leader_election.cc:290] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:48.359095 25547 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:48.359225 25544 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:48.359236 25547 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [term 1 LEADER]: Becoming Leader. State: Replica: f4f4c238ff044376b7a302ac29e1e7e8, State: Running, Role: LEADER
I20250626 20:07:48.359318 25547 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41395 } }
I20250626 20:07:48.359596 25549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f4c238ff044376b7a302ac29e1e7e8. Latest consensus state: current_term: 1 leader_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41395 } } }
I20250626 20:07:48.359584 25548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4c238ff044376b7a302ac29e1e7e8" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 41395 } } }
I20250626 20:07:48.359685 25549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:48.359762 25548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:48.359963 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25477
I20250626 20:07:48.360064 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:48.360178 25555 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:48.360525 25555 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:48.361784 25555 catalog_manager.cc:1349] Generated new cluster ID: a104a9340fc14848950f5eb7b5e19e7d
I20250626 20:07:48.361831 25555 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:48.368868 25555 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:48.369413 25555 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:48.375548 25555 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f4f4c238ff044376b7a302ac29e1e7e8: Generated new TSK 0
I20250626 20:07:48.375679 25555 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:48.381109 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--builtin_ntp_servers=127.20.190.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 20:07:48.448592 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:48.448720 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:48.448746 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:48.449903 25568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 20:07:48.449937 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:48.449968 25568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:48.451264 25568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:48.451439 25568 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:48.451611 25568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:48.454545 25573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.454576 25574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.454704 25576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.454908 25568 server_base.cc:1048] running on GCE node
I20250626 20:07:48.455026 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:48.455193 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:48.456305 25568 hybrid_clock.cc:648] HybridClock initialized: now 1750968468456297 us; error 27 us; skew 500 ppm
I20250626 20:07:48.457393 25568 webserver.cc:469] Webserver started at http://127.20.190.193:46801/ using document root <none> and password file <none>
I20250626 20:07:48.457556 25568 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:48.457597 25568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:48.457700 25568 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:48.458437 25568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f76b1beb11ab45a4ba5956c984185c3d"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.458734 25568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f76b1beb11ab45a4ba5956c984185c3d"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.459661 25568 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:48.460186 25582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.460321 25568 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:48.460397 25568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f76b1beb11ab45a4ba5956c984185c3d"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.460453 25568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:48.481969 25568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:48.482223 25568 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:48.482334 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:48.482545 25568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:48.482812 25568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:48.482844 25568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.482873 25568 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:48.482893 25568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.488902 25568 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:33187
I20250626 20:07:48.488931 25695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:33187 every 8 connection(s)
I20250626 20:07:48.489238 25568 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:48.493619 25696 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41395
I20250626 20:07:48.493693 25696 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:48.493860 25696 heartbeater.cc:507] Master 127.20.190.254:41395 requested a full tablet report, sending...
I20250626 20:07:48.494243 25508 ts_manager.cc:194] Registered new tserver with Master: f76b1beb11ab45a4ba5956c984185c3d (127.20.190.193:33187)
I20250626 20:07:48.494807 25508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:33885
I20250626 20:07:48.497640 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25568
I20250626 20:07:48.497730 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:48.498721 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--builtin_ntp_servers=127.20.190.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 20:07:48.563588 25699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:48.563714 25699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:48.563730 25699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:48.564932 25699 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 20:07:48.564968 25699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:48.564988 25699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:48.566179 25699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.25699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:48.566329 25699 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:48.566476 25699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:48.569569 25704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.569612 25705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.569710 25699 server_base.cc:1048] running on GCE node
W20250626 20:07:48.569612 25707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.569881 25699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:48.570050 25699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:48.571166 25699 hybrid_clock.cc:648] HybridClock initialized: now 1750968468571154 us; error 24 us; skew 500 ppm
I20250626 20:07:48.572118 25699 webserver.cc:469] Webserver started at http://127.20.190.194:35149/ using document root <none> and password file <none>
I20250626 20:07:48.572273 25699 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:48.572309 25699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:48.572399 25699 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:48.573045 25699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5f6acfa2dd6941408817b8a3d2a41bc2"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.573282 25699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5f6acfa2dd6941408817b8a3d2a41bc2"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.574190 25699 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:48.574735 25713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.574874 25699 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:48.574934 25699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5f6acfa2dd6941408817b8a3d2a41bc2"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.574982 25699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:48.591219 25699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:48.591419 25699 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:48.591511 25699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:48.591672 25699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:48.591903 25699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:48.591929 25699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.591957 25699 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:48.591984 25699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.596625 25699 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:36403
I20250626 20:07:48.596681 25826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:36403 every 8 connection(s)
I20250626 20:07:48.596885 25699 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 20:07:48.600279 25827 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41395
I20250626 20:07:48.600356 25827 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:48.600497 25827 heartbeater.cc:507] Master 127.20.190.254:41395 requested a full tablet report, sending...
I20250626 20:07:48.600739 25508 ts_manager.cc:194] Registered new tserver with Master: 5f6acfa2dd6941408817b8a3d2a41bc2 (127.20.190.194:36403)
I20250626 20:07:48.601058 25508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:50289
I20250626 20:07:48.602216 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25699
I20250626 20:07:48.602286 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 20:07:48.603084 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--builtin_ntp_servers=127.20.190.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 20:07:48.669646 25830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:48.669770 25830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:48.669785 25830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:48.670920 25830 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 20:07:48.670954 25830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:48.670974 25830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:48.672111 25830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.25830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:48.672259 25830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:48.672402 25830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:48.675237 25838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.675269 25836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.675304 25830 server_base.cc:1048] running on GCE node
W20250626 20:07:48.675267 25835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.675539 25830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:48.675689 25830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:48.676800 25830 hybrid_clock.cc:648] HybridClock initialized: now 1750968468676792 us; error 22 us; skew 500 ppm
I20250626 20:07:48.677848 25830 webserver.cc:469] Webserver started at http://127.20.190.195:44359/ using document root <none> and password file <none>
I20250626 20:07:48.678059 25830 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:48.678102 25830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:48.678184 25830 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:48.678922 25830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6907ec1b4ab543648019e739910dd7a6"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.679139 25830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6907ec1b4ab543648019e739910dd7a6"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.680001 25830 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:48.680485 25844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.680619 25830 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.680673 25830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6907ec1b4ab543648019e739910dd7a6"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.680727 25830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:48.695050 25830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:48.695253 25830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:48.695344 25830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:48.695505 25830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:48.695749 25830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:48.695777 25830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.695796 25830 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:48.695806 25830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.700796 25830 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:44879
I20250626 20:07:48.700850 25957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:44879 every 8 connection(s)
I20250626 20:07:48.701081 25830 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 20:07:48.704588 25958 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41395
I20250626 20:07:48.704679 25958 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:48.704826 25958 heartbeater.cc:507] Master 127.20.190.254:41395 requested a full tablet report, sending...
I20250626 20:07:48.705111 25508 ts_manager.cc:194] Registered new tserver with Master: 6907ec1b4ab543648019e739910dd7a6 (127.20.190.195:44879)
I20250626 20:07:48.705394 25508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:41413
I20250626 20:07:48.707515 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25830
I20250626 20:07:48.707604 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 20:07:48.708542 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.196:0
--local_ip_for_outbound_sockets=127.20.190.196
--webserver_interface=127.20.190.196
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--builtin_ntp_servers=127.20.190.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 20:07:48.773005 25961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:48.773123 25961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:48.773139 25961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:48.774264 25961 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 20:07:48.774302 25961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:48.774323 25961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.196
I20250626 20:07:48.775537 25961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.190.196
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.25961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.196
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:48.775724 25961 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:48.775892 25961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:48.778854 25969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.778894 25966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.778978 25967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.778980 25961 server_base.cc:1048] running on GCE node
I20250626 20:07:48.779196 25961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:48.779358 25961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:48.780469 25961 hybrid_clock.cc:648] HybridClock initialized: now 1750968468780461 us; error 19 us; skew 500 ppm
I20250626 20:07:48.781387 25961 webserver.cc:469] Webserver started at http://127.20.190.196:34519/ using document root <none> and password file <none>
I20250626 20:07:48.781533 25961 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:48.781570 25961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:48.781661 25961 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:48.782291 25961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cf706adf10a04d6ebd1df830b5b32100"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.782559 25961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cf706adf10a04d6ebd1df830b5b32100"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.783499 25961 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:48.784073 25975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.784205 25961 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:48.784265 25961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cf706adf10a04d6ebd1df830b5b32100"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.784314 25961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:48.798023 25961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:48.798246 25961 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:48.798355 25961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:48.798569 25961 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:48.798807 25961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:48.798844 25961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.798870 25961 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:48.798888 25961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.803439 25961 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.196:40613
I20250626 20:07:48.803493 26088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.196:40613 every 8 connection(s)
I20250626 20:07:48.803694 25961 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 20:07:48.806965 26089 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41395
I20250626 20:07:48.807034 26089 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:48.807180 26089 heartbeater.cc:507] Master 127.20.190.254:41395 requested a full tablet report, sending...
I20250626 20:07:48.807483 25508 ts_manager.cc:194] Registered new tserver with Master: cf706adf10a04d6ebd1df830b5b32100 (127.20.190.196:40613)
I20250626 20:07:48.807829 25508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.196:34591
I20250626 20:07:48.811964 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 25961
I20250626 20:07:48.812042 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 20:07:48.812825 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.197:0
--local_ip_for_outbound_sockets=127.20.190.197
--webserver_interface=127.20.190.197
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--builtin_ntp_servers=127.20.190.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 20:07:48.880548 26092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:48.880699 26092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:48.880717 26092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:48.881877 26092 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 20:07:48.881909 26092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:48.881928 26092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.197
I20250626 20:07:48.883102 26092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.190.197
--webserver_port=0
--tserver_master_addrs=127.20.190.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.26092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.197
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:48.883272 26092 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:48.883425 26092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:48.886381 26100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.886433 26097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:48.886482 26098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:48.886591 26092 server_base.cc:1048] running on GCE node
I20250626 20:07:48.886744 26092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:48.886914 26092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:48.888046 26092 hybrid_clock.cc:648] HybridClock initialized: now 1750968468888029 us; error 26 us; skew 500 ppm
I20250626 20:07:48.889091 26092 webserver.cc:469] Webserver started at http://127.20.190.197:36109/ using document root <none> and password file <none>
I20250626 20:07:48.889257 26092 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:48.889297 26092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:48.889394 26092 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:48.890066 26092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.890318 26092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.891351 26092 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:48.891917 26106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.892058 26092 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:48.892122 26092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506"
format_stamp: "Formatted at 2025-06-26 20:07:48 on dist-test-slave-l47t"
I20250626 20:07:48.892174 26092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:48.903856 26092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:48.904063 26092 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:48.904160 26092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:48.904331 26092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:48.904577 26092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:48.904606 26092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.904636 26092 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:48.904664 26092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:48.909776 26092 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.197:42373
I20250626 20:07:48.909827 26219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.197:42373 every 8 connection(s)
I20250626 20:07:48.910041 26092 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250626 20:07:48.913514 26220 heartbeater.cc:344] Connected to a master server at 127.20.190.254:41395
I20250626 20:07:48.913596 26220 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:48.913743 26220 heartbeater.cc:507] Master 127.20.190.254:41395 requested a full tablet report, sending...
I20250626 20:07:48.914022 25508 ts_manager.cc:194] Registered new tserver with Master: fa72a8f4fbcc4b90a49f3143dbf0c506 (127.20.190.197:42373)
I20250626 20:07:48.914328 25508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.197:41237
I20250626 20:07:48.916414 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 26092
I20250626 20:07:48.916486 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250626 20:07:48.917289 21243 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 20:07:48.922053 25508 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49926:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 20:07:48.929198 25761 tablet_service.cc:1468] Processing CreateTablet for tablet 90304aaf8bea4c9c8ce1a0b3e7a104d9 (DEFAULT_TABLE table=TestTable [id=f09ea34b1cb84ab992487c47309917cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:48.929474 25761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90304aaf8bea4c9c8ce1a0b3e7a104d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:48.929582 25892 tablet_service.cc:1468] Processing CreateTablet for tablet 90304aaf8bea4c9c8ce1a0b3e7a104d9 (DEFAULT_TABLE table=TestTable [id=f09ea34b1cb84ab992487c47309917cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:48.929819 25892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90304aaf8bea4c9c8ce1a0b3e7a104d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:48.930889 26154 tablet_service.cc:1468] Processing CreateTablet for tablet 90304aaf8bea4c9c8ce1a0b3e7a104d9 (DEFAULT_TABLE table=TestTable [id=f09ea34b1cb84ab992487c47309917cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:48.931087 26154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90304aaf8bea4c9c8ce1a0b3e7a104d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:48.932004 26237 tablet_bootstrap.cc:492] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: Bootstrap starting.
I20250626 20:07:48.932705 26237 tablet_bootstrap.cc:654] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:48.932987 26239 tablet_bootstrap.cc:492] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506: Bootstrap starting.
I20250626 20:07:48.933012 26237 log.cc:826] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:48.933081 26238 tablet_bootstrap.cc:492] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: Bootstrap starting.
I20250626 20:07:48.933519 26239 tablet_bootstrap.cc:654] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:48.933543 26238 tablet_bootstrap.cc:654] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:48.933749 26239 log.cc:826] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:48.933759 26238 log.cc:826] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:48.934370 26238 tablet_bootstrap.cc:492] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: No bootstrap required, opened a new log
I20250626 20:07:48.934369 26237 tablet_bootstrap.cc:492] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: No bootstrap required, opened a new log
I20250626 20:07:48.934401 26239 tablet_bootstrap.cc:492] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506: No bootstrap required, opened a new log
I20250626 20:07:48.934433 26238 ts_tablet_manager.cc:1397] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:48.934448 26239 ts_tablet_manager.cc:1397] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:48.934433 26237 ts_tablet_manager.cc:1397] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:48.935465 26239 raft_consensus.cc:357] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:48.935551 26239 raft_consensus.cc:383] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:48.935567 26239 raft_consensus.cc:738] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa72a8f4fbcc4b90a49f3143dbf0c506, State: Initialized, Role: FOLLOWER
I20250626 20:07:48.935635 26239 consensus_queue.cc:260] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:48.935827 26239 ts_tablet_manager.cc:1428] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:48.935838 26220 heartbeater.cc:499] Master 127.20.190.254:41395 was elected leader, sending a full tablet report...
I20250626 20:07:48.935784 26237 raft_consensus.cc:357] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:48.935784 26238 raft_consensus.cc:357] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:48.935899 26238 raft_consensus.cc:383] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:48.935897 26237 raft_consensus.cc:383] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:48.935925 26238 raft_consensus.cc:738] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6907ec1b4ab543648019e739910dd7a6, State: Initialized, Role: FOLLOWER
I20250626 20:07:48.935925 26237 raft_consensus.cc:738] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6acfa2dd6941408817b8a3d2a41bc2, State: Initialized, Role: FOLLOWER
I20250626 20:07:48.936002 26238 consensus_queue.cc:260] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:48.936002 26237 consensus_queue.cc:260] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:48.936201 26238 ts_tablet_manager.cc:1428] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:48.936201 26237 ts_tablet_manager.cc:1428] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:48.936306 25827 heartbeater.cc:499] Master 127.20.190.254:41395 was elected leader, sending a full tablet report...
I20250626 20:07:48.936482 25958 heartbeater.cc:499] Master 127.20.190.254:41395 was elected leader, sending a full tablet report...
W20250626 20:07:48.951485 25959 tablet.cc:2378] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:49.079334 26245 raft_consensus.cc:491] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:49.079421 26245 raft_consensus.cc:513] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:49.079667 26245 leader_election.cc:290] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f6acfa2dd6941408817b8a3d2a41bc2 (127.20.190.194:36403), fa72a8f4fbcc4b90a49f3143dbf0c506 (127.20.190.197:42373)
I20250626 20:07:49.081969 25781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90304aaf8bea4c9c8ce1a0b3e7a104d9" candidate_uuid: "6907ec1b4ab543648019e739910dd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" is_pre_election: true
I20250626 20:07:49.082101 25781 raft_consensus.cc:2466] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6907ec1b4ab543648019e739910dd7a6 in term 0.
I20250626 20:07:49.082296 25845 leader_election.cc:304] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6acfa2dd6941408817b8a3d2a41bc2, 6907ec1b4ab543648019e739910dd7a6; no voters:
I20250626 20:07:49.082456 26245 raft_consensus.cc:2802] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:49.082474 26174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90304aaf8bea4c9c8ce1a0b3e7a104d9" candidate_uuid: "6907ec1b4ab543648019e739910dd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" is_pre_election: true
I20250626 20:07:49.082530 26245 raft_consensus.cc:491] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:49.082584 26245 raft_consensus.cc:3058] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:49.082612 26174 raft_consensus.cc:2466] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6907ec1b4ab543648019e739910dd7a6 in term 0.
I20250626 20:07:49.083227 26245 raft_consensus.cc:513] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:49.083343 26245 leader_election.cc:290] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f6acfa2dd6941408817b8a3d2a41bc2 (127.20.190.194:36403), fa72a8f4fbcc4b90a49f3143dbf0c506 (127.20.190.197:42373)
I20250626 20:07:49.083513 25781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90304aaf8bea4c9c8ce1a0b3e7a104d9" candidate_uuid: "6907ec1b4ab543648019e739910dd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2"
I20250626 20:07:49.083528 26174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90304aaf8bea4c9c8ce1a0b3e7a104d9" candidate_uuid: "6907ec1b4ab543648019e739910dd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506"
I20250626 20:07:49.083592 25781 raft_consensus.cc:3058] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:49.083596 26174 raft_consensus.cc:3058] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:49.084331 26174 raft_consensus.cc:2466] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P fa72a8f4fbcc4b90a49f3143dbf0c506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6907ec1b4ab543648019e739910dd7a6 in term 1.
I20250626 20:07:49.084331 25781 raft_consensus.cc:2466] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6907ec1b4ab543648019e739910dd7a6 in term 1.
I20250626 20:07:49.084491 25845 leader_election.cc:304] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6907ec1b4ab543648019e739910dd7a6, fa72a8f4fbcc4b90a49f3143dbf0c506; no voters:
I20250626 20:07:49.084599 26245 raft_consensus.cc:2802] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:49.084750 26245 raft_consensus.cc:695] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [term 1 LEADER]: Becoming Leader. State: Replica: 6907ec1b4ab543648019e739910dd7a6, State: Running, Role: LEADER
I20250626 20:07:49.084858 26245 consensus_queue.cc:237] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } }
I20250626 20:07:49.085497 25507 catalog_manager.cc:5582] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6907ec1b4ab543648019e739910dd7a6 (127.20.190.195). New cstate: current_term: 1 leader_uuid: "6907ec1b4ab543648019e739910dd7a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6907ec1b4ab543648019e739910dd7a6" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 44879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa72a8f4fbcc4b90a49f3143dbf0c506" member_type: VOTER last_known_addr { host: "127.20.190.197" port: 42373 } health_report { overall_health: UNKNOWN } } }
W20250626 20:07:49.097450 25828 tablet.cc:2378] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:49.106339 21243 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 20:07:49.106791 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5f6acfa2dd6941408817b8a3d2a41bc2 to finish bootstrapping
I20250626 20:07:49.109951 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6907ec1b4ab543648019e739910dd7a6 to finish bootstrapping
I20250626 20:07:49.112445 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fa72a8f4fbcc4b90a49f3143dbf0c506 to finish bootstrapping
I20250626 20:07:49.115020 21243 test_util.cc:276] Using random seed: 496849034
I20250626 20:07:49.119642 21243 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250626 20:07:49.119762 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 26092
I20250626 20:07:49.125140 25781 raft_consensus.cc:1273] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 5f6acfa2dd6941408817b8a3d2a41bc2 [term 1 FOLLOWER]: Refusing update from remote peer 6907ec1b4ab543648019e739910dd7a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 20:07:49.125325 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 20:07:49.125009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.190.197:42373 (local address 127.20.190.195:52909)
0626 20:07:49.125043 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0626 20:07:49.125054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0626 20:07:49.125089 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 20:07:49.125208 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.190.197:42373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":20}
W20250626 20:07:49.125424 25845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.190.197:42373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 20:07:49.125695 26247 consensus_queue.cc:1035] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6acfa2dd6941408817b8a3d2a41bc2" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 20:07:49.125844 25845 consensus_peers.cc:487] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 -> Peer fa72a8f4fbcc4b90a49f3143dbf0c506 (127.20.190.197:42373): Couldn't send request to peer fa72a8f4fbcc4b90a49f3143dbf0c506. Status: Network error: Client connection negotiation failed: client connection to 127.20.190.197:42373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 20:07:49.127272 26257 mvcc.cc:204] Tried to move back new op lower bound from 7171966849534357504 to 7171966849372123136. Current Snapshot: MvccSnapshot[applied={T|T < 7171966849534357504}]
I20250626 20:07:49.127803 26259 mvcc.cc:204] Tried to move back new op lower bound from 7171966849534357504 to 7171966849372123136. Current Snapshot: MvccSnapshot[applied={T|T < 7171966849534357504}]
I20250626 20:07:49.257366 25892 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:49.258328 25761 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:49.264365 25630 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:49.267942 26023 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:49.495476 25696 heartbeater.cc:499] Master 127.20.190.254:41395 was elected leader, sending a full tablet report...
I20250626 20:07:49.808466 26089 heartbeater.cc:499] Master 127.20.190.254:41395 was elected leader, sending a full tablet report...
I20250626 20:07:50.302384 26023 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:50.302740 25630 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:50.302755 25761 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:50.302918 25892 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250626 20:07:51.310317 25845 consensus_peers.cc:487] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 -> Peer fa72a8f4fbcc4b90a49f3143dbf0c506 (127.20.190.197:42373): Couldn't send request to peer fa72a8f4fbcc4b90a49f3143dbf0c506. Status: Network error: Client connection negotiation failed: client connection to 127.20.190.197:42373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 20:07:51.332875 26023 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:51.333405 25761 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:51.334282 25630 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:51.337371 25892 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:52.365510 25630 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:52.365554 26023 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:52.366915 25892 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:52.369843 25761 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:53.397594 25761 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:53.398061 26023 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:53.398149 25892 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:53.399883 25630 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250626 20:07:53.804896 25845 consensus_peers.cc:487] T 90304aaf8bea4c9c8ce1a0b3e7a104d9 P 6907ec1b4ab543648019e739910dd7a6 -> Peer fa72a8f4fbcc4b90a49f3143dbf0c506 (127.20.190.197:42373): Couldn't send request to peer fa72a8f4fbcc4b90a49f3143dbf0c506. Status: Network error: Client connection negotiation failed: client connection to 127.20.190.197:42373: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 20:07:54.415712 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25568
I20250626 20:07:54.419883 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25699
I20250626 20:07:54.424221 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25830
I20250626 20:07:54.429567 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25961
I20250626 20:07:54.432936 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 25477
2025-06-26T20:07:54Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (6202 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (6202 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250626 20:07:54.452257 21243 test_util.cc:276] Using random seed: 502186268
I20250626 20:07:54.452677 21243 ts_itest-base.cc:115] Starting cluster with:
I20250626 20:07:54.452713 21243 ts_itest-base.cc:116] --------------
I20250626 20:07:54.452729 21243 ts_itest-base.cc:117] 1 tablet servers
I20250626 20:07:54.452744 21243 ts_itest-base.cc:118] 1 replicas per TS
I20250626 20:07:54.452767 21243 ts_itest-base.cc:119] --------------
2025-06-26T20:07:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:54Z Disabled control of system clock
I20250626 20:07:54.459491 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:45275
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:45275 with env {}
W20250626 20:07:54.525924 26448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:54.526055 26448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:54.526082 26448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:54.527280 26448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:54.527318 26448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:54.527334 26448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:54.527351 26448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:54.528707 26448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:45275
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:54.528950 26448 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:54.529125 26448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:54.532354 26453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:54.532426 26456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:54.532361 26448 server_base.cc:1048] running on GCE node
W20250626 20:07:54.532436 26454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:54.532761 26448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:54.532951 26448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:54.534073 26448 hybrid_clock.cc:648] HybridClock initialized: now 1750968474534062 us; error 26 us; skew 500 ppm
I20250626 20:07:54.535032 26448 webserver.cc:469] Webserver started at http://127.20.190.254:39709/ using document root <none> and password file <none>
I20250626 20:07:54.535190 26448 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:54.535228 26448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:54.535317 26448 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:54.536000 26448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bab4fe58ad33405ca44ea25475a5fffa"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.536257 26448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bab4fe58ad33405ca44ea25475a5fffa"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.537220 26448 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:54.537739 26462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:54.537917 26448 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:54.537984 26448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bab4fe58ad33405ca44ea25475a5fffa"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.538044 26448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:54.545392 26448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:54.545605 26448 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:54.545703 26448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:54.549073 26448 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:45275
I20250626 20:07:54.549103 26514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:45275 every 8 connection(s)
I20250626 20:07:54.549324 26448 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 20:07:54.549778 26515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:54.551602 26515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa: Bootstrap starting.
I20250626 20:07:54.552062 26515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:54.552237 26515 log.cc:826] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:54.552697 26515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa: No bootstrap required, opened a new log
I20250626 20:07:54.552871 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 26448
I20250626 20:07:54.552944 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 20:07:54.553929 26515 raft_consensus.cc:357] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab4fe58ad33405ca44ea25475a5fffa" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 45275 } }
I20250626 20:07:54.554030 26515 raft_consensus.cc:383] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:54.554059 26515 raft_consensus.cc:738] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab4fe58ad33405ca44ea25475a5fffa, State: Initialized, Role: FOLLOWER
I20250626 20:07:54.554177 26515 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab4fe58ad33405ca44ea25475a5fffa" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 45275 } }
I20250626 20:07:54.554243 26515 raft_consensus.cc:397] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:54.554270 26515 raft_consensus.cc:491] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:54.554306 26515 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:54.554803 26515 raft_consensus.cc:513] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab4fe58ad33405ca44ea25475a5fffa" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 45275 } }
I20250626 20:07:54.554880 26515 leader_election.cc:304] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bab4fe58ad33405ca44ea25475a5fffa; no voters:
I20250626 20:07:54.554991 26515 leader_election.cc:290] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:54.555028 26520 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:54.555151 26520 raft_consensus.cc:695] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [term 1 LEADER]: Becoming Leader. State: Replica: bab4fe58ad33405ca44ea25475a5fffa, State: Running, Role: LEADER
I20250626 20:07:54.555190 26515 sys_catalog.cc:564] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:54.555272 26520 consensus_queue.cc:237] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab4fe58ad33405ca44ea25475a5fffa" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 45275 } }
I20250626 20:07:54.555508 26520 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab4fe58ad33405ca44ea25475a5fffa. Latest consensus state: current_term: 1 leader_uuid: "bab4fe58ad33405ca44ea25475a5fffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab4fe58ad33405ca44ea25475a5fffa" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 45275 } } }
I20250626 20:07:54.555554 26520 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:54.555733 26525 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:54.556176 26522 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bab4fe58ad33405ca44ea25475a5fffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab4fe58ad33405ca44ea25475a5fffa" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 45275 } } }
I20250626 20:07:54.556262 26522 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:54.556505 26525 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:54.557761 26525 catalog_manager.cc:1349] Generated new cluster ID: 6078049effa84b5995118c63f5046d92
I20250626 20:07:54.557804 26525 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:54.567700 26525 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:54.568164 26525 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:54.571113 26525 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bab4fe58ad33405ca44ea25475a5fffa: Generated new TSK 0
I20250626 20:07:54.571231 26525 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:54.574352 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:45275
--builtin_ntp_servers=127.20.190.212:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 20:07:54.641732 26539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:54.641876 26539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:54.641906 26539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:54.643102 26539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:54.643151 26539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:54.644349 26539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:45275
--never_fsync=true
--heap_profile_path=/tmp/kudu.26539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:54.644542 26539 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:54.644778 26539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:54.647869 26545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:54.647929 26547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:54.647850 26544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:54.648216 26539 server_base.cc:1048] running on GCE node
I20250626 20:07:54.648360 26539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:54.648552 26539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:54.649679 26539 hybrid_clock.cc:648] HybridClock initialized: now 1750968474649668 us; error 30 us; skew 500 ppm
I20250626 20:07:54.650835 26539 webserver.cc:469] Webserver started at http://127.20.190.193:42605/ using document root <none> and password file <none>
I20250626 20:07:54.650992 26539 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:54.651027 26539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:54.651126 26539 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:54.651796 26539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7a580a1e7d224de881093b04118c88d4"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.652060 26539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7a580a1e7d224de881093b04118c88d4"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.653036 26539 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:54.653614 26553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:54.653756 26539 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:54.653813 26539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7a580a1e7d224de881093b04118c88d4"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.653859 26539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:54.664165 26539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:54.664377 26539 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:54.664474 26539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:54.664651 26539 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:54.664927 26539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:54.664956 26539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:54.664974 26539 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:54.664985 26539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:54.671087 26539 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:42797
I20250626 20:07:54.671144 26666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:42797 every 8 connection(s)
I20250626 20:07:54.671376 26539 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 20:07:54.675487 26667 heartbeater.cc:344] Connected to a master server at 127.20.190.254:45275
I20250626 20:07:54.675572 26667 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:54.675778 26667 heartbeater.cc:507] Master 127.20.190.254:45275 requested a full tablet report, sending...
I20250626 20:07:54.676205 26479 ts_manager.cc:194] Registered new tserver with Master: 7a580a1e7d224de881093b04118c88d4 (127.20.190.193:42797)
I20250626 20:07:54.676741 26479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:54911
I20250626 20:07:54.678462 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 26539
I20250626 20:07:54.678591 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750968436877382-21243-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 20:07:54.679431 21243 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 20:07:54.684528 26479 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34210:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 20:07:54.689714 26601 tablet_service.cc:1468] Processing CreateTablet for tablet 9113022f9a874143951e98e4fc24d45e (DEFAULT_TABLE table=TestTable [id=46ea76c4cb844131b810af109a0597bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:54.689936 26601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9113022f9a874143951e98e4fc24d45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:54.691641 26677 tablet_bootstrap.cc:492] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4: Bootstrap starting.
I20250626 20:07:54.692108 26677 tablet_bootstrap.cc:654] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:54.692364 26677 log.cc:826] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:54.692921 26677 tablet_bootstrap.cc:492] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4: No bootstrap required, opened a new log
I20250626 20:07:54.692992 26677 ts_tablet_manager.cc:1397] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:54.693840 26677 raft_consensus.cc:357] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a580a1e7d224de881093b04118c88d4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42797 } }
I20250626 20:07:54.693935 26677 raft_consensus.cc:383] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:54.693959 26677 raft_consensus.cc:738] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a580a1e7d224de881093b04118c88d4, State: Initialized, Role: FOLLOWER
I20250626 20:07:54.694011 26677 consensus_queue.cc:260] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a580a1e7d224de881093b04118c88d4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42797 } }
I20250626 20:07:54.694049 26677 raft_consensus.cc:397] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:54.694067 26677 raft_consensus.cc:491] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:54.694092 26677 raft_consensus.cc:3058] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:54.694482 26677 raft_consensus.cc:513] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a580a1e7d224de881093b04118c88d4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42797 } }
I20250626 20:07:54.694592 26677 leader_election.cc:304] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a580a1e7d224de881093b04118c88d4; no voters:
I20250626 20:07:54.694700 26677 leader_election.cc:290] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:54.694743 26679 raft_consensus.cc:2802] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:54.694851 26677 ts_tablet_manager.cc:1428] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:54.694864 26667 heartbeater.cc:499] Master 127.20.190.254:45275 was elected leader, sending a full tablet report...
I20250626 20:07:54.694887 26679 raft_consensus.cc:695] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7a580a1e7d224de881093b04118c88d4, State: Running, Role: LEADER
I20250626 20:07:54.695078 26679 consensus_queue.cc:237] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a580a1e7d224de881093b04118c88d4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42797 } }
I20250626 20:07:54.695710 26479 catalog_manager.cc:5582] T 9113022f9a874143951e98e4fc24d45e P 7a580a1e7d224de881093b04118c88d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a580a1e7d224de881093b04118c88d4 (127.20.190.193). New cstate: current_term: 1 leader_uuid: "7a580a1e7d224de881093b04118c88d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a580a1e7d224de881093b04118c88d4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42797 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:54.700099 21243 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 20:07:54.700445 21243 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7a580a1e7d224de881093b04118c88d4 to finish bootstrapping
I20250626 20:07:54.785967 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 26539
I20250626 20:07:54.790475 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 26448
2025-06-26T20:07:54Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (355 ms)
[----------] 1 test from ListTableCliSimpleParamTest (355 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250626 20:07:54.807727 21243 test_util.cc:276] Using random seed: 502541739
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (1 ms)
[----------] 1 test from ListTableCliParamTest (1 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-26T20:07:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T20:07:54Z Disabled control of system clock
I20250626 20:07:54.815066 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:32957
--webserver_interface=127.20.190.254
--webserver_port=0
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:32957 with env {}
W20250626 20:07:54.881052 26701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:54.881186 26701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:54.881201 26701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:54.882393 26701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:54.882431 26701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:54.882442 26701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:54.882453 26701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:54.883713 26701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:32957
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:32957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:54.883879 26701 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:54.884066 26701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:54.887177 26707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:54.887181 26709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:54.887187 26706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:54.887337 26701 server_base.cc:1048] running on GCE node
I20250626 20:07:54.887522 26701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:54.887719 26701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:54.888878 26701 hybrid_clock.cc:648] HybridClock initialized: now 1750968474888870 us; error 32 us; skew 500 ppm
I20250626 20:07:54.892736 26701 webserver.cc:469] Webserver started at http://127.20.190.254:46873/ using document root <none> and password file <none>
I20250626 20:07:54.892936 26701 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:54.892989 26701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:54.893090 26701 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:54.893944 26701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/instance:
uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.894165 26701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal/instance:
uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.895169 26701 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 20:07:54.895702 26715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:54.895853 26701 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:54.895915 26701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452"
format_stamp: "Formatted at 2025-06-26 20:07:54 on dist-test-slave-l47t"
I20250626 20:07:54.895975 26701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:54.909493 26701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:54.909718 26701 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:54.909796 26701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:54.912947 26701 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:32957
I20250626 20:07:54.912977 26767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:32957 every 8 connection(s)
I20250626 20:07:54.913208 26701 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb
I20250626 20:07:54.913679 26768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:54.915560 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452: Bootstrap starting.
I20250626 20:07:54.916052 26768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:54.916257 26768 log.cc:826] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:54.916738 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452: No bootstrap required, opened a new log
I20250626 20:07:54.917675 26768 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:54.917799 26768 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:54.917820 26768 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb8c06e4d444ba7a67ea4b38c0b5452, State: Initialized, Role: FOLLOWER
I20250626 20:07:54.917936 26768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:54.917990 26768 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:54.918010 26768 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:54.918049 26768 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:54.918370 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 26701
I20250626 20:07:54.918467 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal/instance
I20250626 20:07:54.918743 26768 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:54.918861 26768 leader_election.cc:304] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb8c06e4d444ba7a67ea4b38c0b5452; no voters:
I20250626 20:07:54.919008 26768 leader_election.cc:290] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:54.919076 26771 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:54.919194 26768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:54.919247 26771 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb8c06e4d444ba7a67ea4b38c0b5452, State: Running, Role: LEADER
I20250626 20:07:54.919481 26771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:54.919804 26772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } } }
I20250626 20:07:54.919935 26772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:54.920113 26774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cb8c06e4d444ba7a67ea4b38c0b5452. Latest consensus state: current_term: 1 leader_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb8c06e4d444ba7a67ea4b38c0b5452" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } } }
I20250626 20:07:54.920217 26774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:54.920351 26782 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:54.920756 26782 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:54.922407 26782 catalog_manager.cc:1349] Generated new cluster ID: 57833ab78f7e4fce9156a8194bc462e5
I20250626 20:07:54.922484 26782 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:54.933323 26782 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:54.933949 26782 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:54.941531 26782 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1cb8c06e4d444ba7a67ea4b38c0b5452: Generated new TSK 0
I20250626 20:07:54.941665 26782 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 20:07:54.951637 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:0
--local_ip_for_outbound_sockets=127.20.190.193
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:32957
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250626 20:07:55.036248 26792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:55.036376 26792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:55.036392 26792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:55.037568 26792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:55.037613 26792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:55.039041 26792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=0
--tserver_master_addrs=127.20.190.254:32957
--never_fsync=true
--heap_profile_path=/tmp/kudu.26792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:55.039217 26792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:55.039405 26792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:55.042805 26798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.042819 26800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.043010 26797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:55.043088 26792 server_base.cc:1048] running on GCE node
I20250626 20:07:55.043231 26792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:55.043406 26792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:55.044536 26792 hybrid_clock.cc:648] HybridClock initialized: now 1750968475044526 us; error 26 us; skew 500 ppm
I20250626 20:07:55.045535 26792 webserver.cc:469] Webserver started at http://127.20.190.193:38235/ using document root <none> and password file <none>
I20250626 20:07:55.045696 26792 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:55.045733 26792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:55.045827 26792 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:55.046495 26792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/instance:
uuid: "d40be74472824d04ba807bfab764b6e4"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.046790 26792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal/instance:
uuid: "d40be74472824d04ba807bfab764b6e4"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.047814 26792 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:55.048449 26806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.048658 26792 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:55.048755 26792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
uuid: "d40be74472824d04ba807bfab764b6e4"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.048818 26792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:55.065551 26792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:55.065809 26792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:55.065912 26792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:55.066090 26792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:55.066337 26792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:55.066370 26792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.066392 26792 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:55.066411 26792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.071676 26792 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:42071
I20250626 20:07:55.071745 26919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:42071 every 8 connection(s)
I20250626 20:07:55.071954 26792 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb
I20250626 20:07:55.075632 26920 heartbeater.cc:344] Connected to a master server at 127.20.190.254:32957
I20250626 20:07:55.075704 26920 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:55.075852 26920 heartbeater.cc:507] Master 127.20.190.254:32957 requested a full tablet report, sending...
I20250626 20:07:55.076251 26732 ts_manager.cc:194] Registered new tserver with Master: d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071)
I20250626 20:07:55.076296 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 26792
I20250626 20:07:55.076378 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal/instance
I20250626 20:07:55.076983 26732 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:36051
I20250626 20:07:55.077458 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:0
--local_ip_for_outbound_sockets=127.20.190.194
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:32957
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250626 20:07:55.142673 26923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:55.142802 26923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:55.142829 26923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:55.144042 26923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:55.144091 26923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:55.145452 26923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=0
--tserver_master_addrs=127.20.190.254:32957
--never_fsync=true
--heap_profile_path=/tmp/kudu.26923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:55.145655 26923 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:55.145885 26923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:55.149017 26931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.149050 26929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:55.149154 26923 server_base.cc:1048] running on GCE node
W20250626 20:07:55.149209 26928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:55.149333 26923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:55.149479 26923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:55.150564 26923 hybrid_clock.cc:648] HybridClock initialized: now 1750968475150549 us; error 24 us; skew 500 ppm
I20250626 20:07:55.151475 26923 webserver.cc:469] Webserver started at http://127.20.190.194:36045/ using document root <none> and password file <none>
I20250626 20:07:55.151624 26923 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:55.151661 26923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:55.151764 26923 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:55.152407 26923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/instance:
uuid: "3fa147f5bbf1479cb6393c93fc448716"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.152652 26923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal/instance:
uuid: "3fa147f5bbf1479cb6393c93fc448716"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.153743 26923 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:55.154300 26937 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.154446 26923 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:55.154507 26923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
uuid: "3fa147f5bbf1479cb6393c93fc448716"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.154585 26923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:55.165886 26923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:55.166090 26923 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:55.166184 26923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:55.166339 26923 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:55.166584 26923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:55.166620 26923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.166652 26923 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:55.166683 26923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.171905 26923 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:37415
I20250626 20:07:55.171972 27050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:37415 every 8 connection(s)
I20250626 20:07:55.172170 26923 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb
I20250626 20:07:55.175788 27051 heartbeater.cc:344] Connected to a master server at 127.20.190.254:32957
I20250626 20:07:55.175854 27051 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:55.176010 27051 heartbeater.cc:507] Master 127.20.190.254:32957 requested a full tablet report, sending...
I20250626 20:07:55.176277 26731 ts_manager.cc:194] Registered new tserver with Master: 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415)
I20250626 20:07:55.176576 26731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:32811
I20250626 20:07:55.181033 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 26923
I20250626 20:07:55.181118 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal/instance
I20250626 20:07:55.181946 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:0
--local_ip_for_outbound_sockets=127.20.190.195
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:32957
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250626 20:07:55.247689 27054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:55.247828 27054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:55.247845 27054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:55.249096 27054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:55.249131 27054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:55.250247 27054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=0
--tserver_master_addrs=127.20.190.254:32957
--never_fsync=true
--heap_profile_path=/tmp/kudu.27054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:55.250391 27054 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:55.250551 27054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:55.253455 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.253464 27062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.253475 27060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:55.253626 27054 server_base.cc:1048] running on GCE node
I20250626 20:07:55.253757 27054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:55.253921 27054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:55.255038 27054 hybrid_clock.cc:648] HybridClock initialized: now 1750968475255020 us; error 33 us; skew 500 ppm
I20250626 20:07:55.256034 27054 webserver.cc:469] Webserver started at http://127.20.190.195:39467/ using document root <none> and password file <none>
I20250626 20:07:55.256191 27054 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:55.256227 27054 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:55.256325 27054 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:55.256966 27054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/instance:
uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.257212 27054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal/instance:
uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.258273 27054 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:55.258972 27068 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.259126 27054 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.259186 27054 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.259238 27054 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:55.268846 27054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:55.269079 27054 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:55.269171 27054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:55.269340 27054 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:55.269558 27054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 20:07:55.269585 27054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.269610 27054 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 20:07:55.269627 27054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.274224 27054 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:37755
I20250626 20:07:55.274279 27181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:37755 every 8 connection(s)
I20250626 20:07:55.274477 27054 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb
I20250626 20:07:55.275591 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 27054
I20250626 20:07:55.275686 21243 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal/instance
I20250626 20:07:55.278066 27182 heartbeater.cc:344] Connected to a master server at 127.20.190.254:32957
I20250626 20:07:55.278156 27182 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:55.278328 27182 heartbeater.cc:507] Master 127.20.190.254:32957 requested a full tablet report, sending...
I20250626 20:07:55.278638 26731 ts_manager.cc:194] Registered new tserver with Master: 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755)
I20250626 20:07:55.278988 26731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:38691
I20250626 20:07:55.287061 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:55.296627 21243 test_util.cc:276] Using random seed: 503030642
I20250626 20:07:55.301729 26731 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55524:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 20:07:55.301940 26731 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 20:07:55.307235 26854 tablet_service.cc:1468] Processing CreateTablet for tablet 80cdb32ba31540d1997f0990a3584aae (DEFAULT_TABLE table=pre_rebuild [id=3ac45e9d4abe4514bc5ac71ba3673d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:55.307492 26854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cdb32ba31540d1997f0990a3584aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:55.307862 26985 tablet_service.cc:1468] Processing CreateTablet for tablet 80cdb32ba31540d1997f0990a3584aae (DEFAULT_TABLE table=pre_rebuild [id=3ac45e9d4abe4514bc5ac71ba3673d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:55.307979 27116 tablet_service.cc:1468] Processing CreateTablet for tablet 80cdb32ba31540d1997f0990a3584aae (DEFAULT_TABLE table=pre_rebuild [id=3ac45e9d4abe4514bc5ac71ba3673d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:55.308125 26985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cdb32ba31540d1997f0990a3584aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:55.308175 27116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cdb32ba31540d1997f0990a3584aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:55.309978 27201 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Bootstrap starting.
I20250626 20:07:55.310214 27202 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Bootstrap starting.
I20250626 20:07:55.310468 27201 tablet_bootstrap.cc:654] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:55.310509 27203 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Bootstrap starting.
I20250626 20:07:55.310758 27201 log.cc:826] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:55.310798 27202 tablet_bootstrap.cc:654] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:55.311060 27203 tablet_bootstrap.cc:654] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:55.311049 27202 log.cc:826] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:55.311303 27203 log.cc:826] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:55.311394 27201 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: No bootstrap required, opened a new log
I20250626 20:07:55.311457 27201 ts_tablet_manager.cc:1397] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 20:07:55.311661 27202 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: No bootstrap required, opened a new log
I20250626 20:07:55.311748 27202 ts_tablet_manager.cc:1397] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:55.312721 27201 raft_consensus.cc:357] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.312846 27201 raft_consensus.cc:383] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:55.312871 27201 raft_consensus.cc:738] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906f1c6e03ea4fe38483fa3f6fdb44fb, State: Initialized, Role: FOLLOWER
I20250626 20:07:55.312963 27201 consensus_queue.cc:260] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.312989 27202 raft_consensus.cc:357] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.313099 27202 raft_consensus.cc:383] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:55.313134 27202 raft_consensus.cc:738] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40be74472824d04ba807bfab764b6e4, State: Initialized, Role: FOLLOWER
I20250626 20:07:55.313182 27201 ts_tablet_manager.cc:1428] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 20:07:55.313207 27202 consensus_queue.cc:260] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.313284 27182 heartbeater.cc:499] Master 127.20.190.254:32957 was elected leader, sending a full tablet report...
I20250626 20:07:55.313423 27202 ts_tablet_manager.cc:1428] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 20:07:55.313449 26920 heartbeater.cc:499] Master 127.20.190.254:32957 was elected leader, sending a full tablet report...
I20250626 20:07:55.313970 27203 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: No bootstrap required, opened a new log
I20250626 20:07:55.314033 27203 ts_tablet_manager.cc:1397] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250626 20:07:55.314996 27203 raft_consensus.cc:357] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.315109 27203 raft_consensus.cc:383] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:55.315138 27203 raft_consensus.cc:738] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa147f5bbf1479cb6393c93fc448716, State: Initialized, Role: FOLLOWER
I20250626 20:07:55.315227 27203 consensus_queue.cc:260] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.315405 27203 ts_tablet_manager.cc:1428] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:55.315423 27051 heartbeater.cc:499] Master 127.20.190.254:32957 was elected leader, sending a full tablet report...
W20250626 20:07:55.322360 26921 tablet.cc:2378] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:55.422608 27052 tablet.cc:2378] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 20:07:55.524930 27183 tablet.cc:2378] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 20:07:55.571779 27207 raft_consensus.cc:491] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:55.571895 27207 raft_consensus.cc:513] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.572175 27207 leader_election.cc:290] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415), d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071)
I20250626 20:07:55.574879 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fa147f5bbf1479cb6393c93fc448716" is_pre_election: true
I20250626 20:07:55.575014 27005 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 0.
I20250626 20:07:55.575215 27069 leader_election.cc:304] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa147f5bbf1479cb6393c93fc448716, 906f1c6e03ea4fe38483fa3f6fdb44fb; no voters:
I20250626 20:07:55.575423 26874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40be74472824d04ba807bfab764b6e4" is_pre_election: true
I20250626 20:07:55.575485 27207 raft_consensus.cc:2802] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:55.575551 27207 raft_consensus.cc:491] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:55.575556 26874 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 0.
I20250626 20:07:55.575577 27207 raft_consensus.cc:3058] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:55.576165 27207 raft_consensus.cc:513] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.576254 27207 leader_election.cc:290] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 1 election: Requested vote from peers 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415), d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071)
I20250626 20:07:55.576447 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fa147f5bbf1479cb6393c93fc448716"
I20250626 20:07:55.576443 26874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40be74472824d04ba807bfab764b6e4"
I20250626 20:07:55.576503 27005 raft_consensus.cc:3058] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:55.576506 26874 raft_consensus.cc:3058] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:55.577055 26874 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 1.
I20250626 20:07:55.577224 27069 leader_election.cc:304] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906f1c6e03ea4fe38483fa3f6fdb44fb, d40be74472824d04ba807bfab764b6e4; no voters:
I20250626 20:07:55.577314 27207 raft_consensus.cc:2802] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:55.577332 27005 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 1.
I20250626 20:07:55.577447 27207 raft_consensus.cc:695] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 LEADER]: Becoming Leader. State: Replica: 906f1c6e03ea4fe38483fa3f6fdb44fb, State: Running, Role: LEADER
I20250626 20:07:55.577539 27207 consensus_queue.cc:237] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:55.578281 26732 catalog_manager.cc:5582] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195). New cstate: current_term: 1 leader_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:55.591238 27005 raft_consensus.cc:1273] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Refusing update from remote peer 906f1c6e03ea4fe38483fa3f6fdb44fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 20:07:55.591238 26874 raft_consensus.cc:1273] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Refusing update from remote peer 906f1c6e03ea4fe38483fa3f6fdb44fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 20:07:55.591496 27207 consensus_queue.cc:1035] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:55.591619 27211 consensus_queue.cc:1035] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:55.594049 27218 mvcc.cc:204] Tried to move back new op lower bound from 7171966876019339264 to 7171966875965988864. Current Snapshot: MvccSnapshot[applied={T|T < 7171966876019339264}]
I20250626 20:07:55.594442 27220 mvcc.cc:204] Tried to move back new op lower bound from 7171966876019339264 to 7171966875965988864. Current Snapshot: MvccSnapshot[applied={T|T < 7171966876019339264}]
I20250626 20:07:55.595053 27219 mvcc.cc:204] Tried to move back new op lower bound from 7171966876019339264 to 7171966875965988864. Current Snapshot: MvccSnapshot[applied={T|T < 7171966876019339264}]
I20250626 20:07:55.743083 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 26701
W20250626 20:07:55.815740 27239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:55.815871 27239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:55.817013 27239 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 20:07:55.832985 27239 minidump.cc:252] Setting minidump size limit to 20M
I20250626 20:07:55.833184 27239 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:55.833318 27239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:55.835117 27260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.835197 27261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:55.835407 27263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:55.835572 27239 server_base.cc:1048] running on GCE node
I20250626 20:07:55.835788 27239 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250626 20:07:55.835868 27239 hybrid_clock.cc:648] HybridClock initialized: now 1750968475835868 us; error 28834 us; skew 500 ppm
I20250626 20:07:55.836632 27239 webserver.cc:469] Webserver started at http://0.0.0.0:33729/ using document root <none> and password file <none>
I20250626 20:07:55.836804 27239 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:55.836867 27239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:55.836956 27239 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 20:07:55.837803 27239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/instance:
uuid: "453db716f51a4ec780dd8187c8dbf2e5"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.838161 27239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal/instance:
uuid: "453db716f51a4ec780dd8187c8dbf2e5"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.839484 27239 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20250626 20:07:55.840111 27268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:55.840260 27239 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250626 20:07:55.840330 27239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
uuid: "453db716f51a4ec780dd8187c8dbf2e5"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:55.840385 27239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:55.932919 27239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:55.933164 27239 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:55.933257 27239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:55.933651 27239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:55.935554 27239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Bootstrap starting.
I20250626 20:07:55.936009 27239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:55.936445 27239 log.cc:826] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:55.936939 27239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: No bootstrap required, opened a new log
I20250626 20:07:55.937731 27239 raft_consensus.cc:357] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER }
I20250626 20:07:55.937848 27239 raft_consensus.cc:383] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:55.937867 27239 raft_consensus.cc:738] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453db716f51a4ec780dd8187c8dbf2e5, State: Initialized, Role: FOLLOWER
I20250626 20:07:55.937938 27239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER }
I20250626 20:07:55.937980 27239 raft_consensus.cc:397] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:55.938000 27239 raft_consensus.cc:491] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:55.938027 27239 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:55.938441 27239 raft_consensus.cc:513] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER }
I20250626 20:07:55.938537 27239 leader_election.cc:304] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453db716f51a4ec780dd8187c8dbf2e5; no voters:
I20250626 20:07:55.938647 27239 leader_election.cc:290] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 20:07:55.938695 27275 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:55.938824 27275 raft_consensus.cc:695] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 LEADER]: Becoming Leader. State: Replica: 453db716f51a4ec780dd8187c8dbf2e5, State: Running, Role: LEADER
I20250626 20:07:55.938895 27275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER }
I20250626 20:07:55.939208 27277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453db716f51a4ec780dd8187c8dbf2e5. Latest consensus state: current_term: 1 leader_uuid: "453db716f51a4ec780dd8187c8dbf2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER } }
I20250626 20:07:55.939229 27276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "453db716f51a4ec780dd8187c8dbf2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER } }
I20250626 20:07:55.939405 27276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:55.939353 27277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:55.940352 27239 tablet_replica.cc:331] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: stopping tablet replica
I20250626 20:07:55.940419 27239 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 LEADER]: Raft consensus shutting down.
I20250626 20:07:55.940465 27239 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 20:07:55.940771 27239 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 20:07:55.940850 27239 acceptor_pool.cc:194] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 20:07:55.941483 27239 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250626 20:07:55.946907 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 26792
I20250626 20:07:55.951385 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 26923
I20250626 20:07:55.955318 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 27054
I20250626 20:07:55.960067 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:32957
--webserver_interface=127.20.190.254
--webserver_port=46873
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.190.254:32957 with env {}
W20250626 20:07:56.027060 27281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:56.027180 27281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:56.027195 27281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:56.028460 27281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 20:07:56.028498 27281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:56.028512 27281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 20:07:56.028525 27281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 20:07:56.029855 27281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.190.254:32957
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.190.254:32957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.190.254
--webserver_port=46873
--never_fsync=true
--heap_profile_path=/tmp/kudu.27281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:56.030030 27281 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:56.030210 27281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:56.033053 27287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.033061 27289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.033142 27286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:56.033336 27281 server_base.cc:1048] running on GCE node
I20250626 20:07:56.033505 27281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:56.033695 27281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:56.034811 27281 hybrid_clock.cc:648] HybridClock initialized: now 1750968476034798 us; error 24 us; skew 500 ppm
I20250626 20:07:56.035849 27281 webserver.cc:469] Webserver started at http://127.20.190.254:46873/ using document root <none> and password file <none>
I20250626 20:07:56.036011 27281 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:56.036063 27281 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:56.036998 27281 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:56.037468 27295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.037607 27281 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.037667 27281 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
uuid: "453db716f51a4ec780dd8187c8dbf2e5"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:56.037873 27281 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:56.055896 27281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:56.056107 27281 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:56.056205 27281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:56.059563 27281 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.254:32957
I20250626 20:07:56.059602 27347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.254:32957 every 8 connection(s)
I20250626 20:07:56.059844 27281 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb
I20250626 20:07:56.060668 27348 sys_catalog.cc:263] Verifying existing consensus state
I20250626 20:07:56.062124 27348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Bootstrap starting.
I20250626 20:07:56.062984 27348 log.cc:826] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:56.063479 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 27281
I20250626 20:07:56.063707 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.193:42071
--local_ip_for_outbound_sockets=127.20.190.193
--tserver_master_addrs=127.20.190.254:32957
--webserver_port=38235
--webserver_interface=127.20.190.193
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 20:07:56.064093 27348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:56.064303 27348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Bootstrap complete.
I20250626 20:07:56.065407 27348 raft_consensus.cc:357] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:56.065512 27348 raft_consensus.cc:738] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453db716f51a4ec780dd8187c8dbf2e5, State: Initialized, Role: FOLLOWER
I20250626 20:07:56.065609 27348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:56.065677 27348 raft_consensus.cc:397] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 20:07:56.065718 27348 raft_consensus.cc:491] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 20:07:56.065776 27348 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:56.066222 27348 raft_consensus.cc:513] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:56.066296 27348 leader_election.cc:304] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453db716f51a4ec780dd8187c8dbf2e5; no voters:
I20250626 20:07:56.066406 27348 leader_election.cc:290] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 20:07:56.066463 27352 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:56.066573 27352 raft_consensus.cc:695] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [term 2 LEADER]: Becoming Leader. State: Replica: 453db716f51a4ec780dd8187c8dbf2e5, State: Running, Role: LEADER
I20250626 20:07:56.066617 27348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 20:07:56.066648 27352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } }
I20250626 20:07:56.066841 27352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453db716f51a4ec780dd8187c8dbf2e5. Latest consensus state: current_term: 2 leader_uuid: "453db716f51a4ec780dd8187c8dbf2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } } }
I20250626 20:07:56.066900 27352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:56.066856 27353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "453db716f51a4ec780dd8187c8dbf2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453db716f51a4ec780dd8187c8dbf2e5" member_type: VOTER last_known_addr { host: "127.20.190.254" port: 32957 } } }
I20250626 20:07:56.067058 27353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5 [sys.catalog]: This master's current role is: LEADER
I20250626 20:07:56.067058 27356 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 20:07:56.067685 27356 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=9fb8f5a3475048f59fccbae53d653d3a]
I20250626 20:07:56.067996 27356 tablet_loader.cc:96] loaded metadata for tablet 80cdb32ba31540d1997f0990a3584aae (table pre_rebuild [id=9fb8f5a3475048f59fccbae53d653d3a])
I20250626 20:07:56.068113 27356 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 20:07:56.069409 27356 catalog_manager.cc:1349] Generated new cluster ID: fc3ee0def23847d9bfa8cdf4ca9e59f8
I20250626 20:07:56.069448 27356 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 20:07:56.084007 27356 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 20:07:56.084564 27356 catalog_manager.cc:1506] Loading token signing keys...
I20250626 20:07:56.093189 27356 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Generated new TSK 0
I20250626 20:07:56.093305 27356 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 20:07:56.136548 27351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:56.136678 27351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:56.136694 27351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:56.137938 27351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:56.137987 27351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.193
I20250626 20:07:56.139351 27351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.193:42071
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.190.193
--webserver_port=38235
--tserver_master_addrs=127.20.190.254:32957
--never_fsync=true
--heap_profile_path=/tmp/kudu.27351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.193
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:56.139511 27351 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:56.139698 27351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:56.142762 27378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.142762 27376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.142974 27375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:56.143254 27351 server_base.cc:1048] running on GCE node
I20250626 20:07:56.143374 27351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:56.143538 27351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:56.144668 27351 hybrid_clock.cc:648] HybridClock initialized: now 1750968476144655 us; error 43 us; skew 500 ppm
I20250626 20:07:56.145717 27351 webserver.cc:469] Webserver started at http://127.20.190.193:38235/ using document root <none> and password file <none>
I20250626 20:07:56.145882 27351 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:56.145932 27351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:56.146943 27351 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:56.147469 27384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.147675 27351 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:56.147751 27351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
uuid: "d40be74472824d04ba807bfab764b6e4"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:56.148041 27351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:56.162297 27351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:56.162577 27351 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:56.162685 27351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:56.162878 27351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:56.163235 27391 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250626 20:07:56.163906 27351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 20:07:56.163957 27351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:56.163985 27351 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 20:07:56.164616 27351 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 20:07:56.164659 27351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:56.164698 27391 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Bootstrap starting.
I20250626 20:07:56.171340 27351 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.193:42071
I20250626 20:07:56.171492 27498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.193:42071 every 8 connection(s)
I20250626 20:07:56.171691 27351 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb
I20250626 20:07:56.177866 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 27351
I20250626 20:07:56.177927 27499 heartbeater.cc:344] Connected to a master server at 127.20.190.254:32957
I20250626 20:07:56.178020 27499 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:56.178165 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.194:37415
--local_ip_for_outbound_sockets=127.20.190.194
--tserver_master_addrs=127.20.190.254:32957
--webserver_port=36045
--webserver_interface=127.20.190.194
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 20:07:56.178196 27499 heartbeater.cc:507] Master 127.20.190.254:32957 requested a full tablet report, sending...
I20250626 20:07:56.178418 27391 log.cc:826] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:56.179121 27312 ts_manager.cc:194] Registered new tserver with Master: d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071)
I20250626 20:07:56.180048 27312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.193:35679
I20250626 20:07:56.218047 27391 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=218 overwritten=0 applied=218 ignored=0} inserts{seen=10850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:56.218379 27391 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Bootstrap complete.
I20250626 20:07:56.219465 27391 ts_tablet_manager.cc:1397] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Time spent bootstrapping tablet: real 0.055s user 0.042s sys 0.012s
I20250626 20:07:56.220376 27391 raft_consensus.cc:357] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:56.220580 27391 raft_consensus.cc:738] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40be74472824d04ba807bfab764b6e4, State: Initialized, Role: FOLLOWER
I20250626 20:07:56.220666 27391 consensus_queue.cc:260] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:56.220902 27391 ts_tablet_manager.cc:1428] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 20:07:56.220953 27499 heartbeater.cc:499] Master 127.20.190.254:32957 was elected leader, sending a full tablet report...
W20250626 20:07:56.262377 27504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:56.262501 27504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:56.262535 27504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:56.263720 27504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:56.263764 27504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.194
I20250626 20:07:56.264979 27504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.194:37415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.190.194
--webserver_port=36045
--tserver_master_addrs=127.20.190.254:32957
--never_fsync=true
--heap_profile_path=/tmp/kudu.27504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.194
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:56.265172 27504 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:56.265364 27504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:56.268543 27510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:56.268604 27504 server_base.cc:1048] running on GCE node
W20250626 20:07:56.268565 27513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.268618 27511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:56.268834 27504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:56.269023 27504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:56.270165 27504 hybrid_clock.cc:648] HybridClock initialized: now 1750968476270145 us; error 32 us; skew 500 ppm
I20250626 20:07:56.271131 27504 webserver.cc:469] Webserver started at http://127.20.190.194:36045/ using document root <none> and password file <none>
I20250626 20:07:56.271296 27504 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:56.271348 27504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:56.272281 27504 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:56.272754 27519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.272904 27504 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.272964 27504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
uuid: "3fa147f5bbf1479cb6393c93fc448716"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:56.273149 27504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:56.281687 27504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:56.281909 27504 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:56.282011 27504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:56.282184 27504 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:56.282495 27526 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250626 20:07:56.283159 27504 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 20:07:56.283191 27504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:56.283219 27504 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 20:07:56.283684 27504 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 20:07:56.283715 27504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.283778 27526 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Bootstrap starting.
I20250626 20:07:56.289567 27504 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.194:37415
I20250626 20:07:56.289608 27633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.194:37415 every 8 connection(s)
I20250626 20:07:56.289853 27504 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb
I20250626 20:07:56.294449 27634 heartbeater.cc:344] Connected to a master server at 127.20.190.254:32957
I20250626 20:07:56.294610 27634 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:56.294817 27634 heartbeater.cc:507] Master 127.20.190.254:32957 requested a full tablet report, sending...
I20250626 20:07:56.295359 27312 ts_manager.cc:194] Registered new tserver with Master: 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415)
I20250626 20:07:56.295846 27312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.194:58439
I20250626 20:07:56.296289 27526 log.cc:826] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:56.299707 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 27504
I20250626 20:07:56.299914 21243 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasksd6vjS/build/release/bin/kudu
/tmp/dist-test-tasksd6vjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.190.195:37755
--local_ip_for_outbound_sockets=127.20.190.195
--tserver_master_addrs=127.20.190.254:32957
--webserver_port=39467
--webserver_interface=127.20.190.195
--builtin_ntp_servers=127.20.190.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 20:07:56.337997 27526 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Bootstrap replayed 1/1 log segments. Stats: ops{read=218 overwritten=0 applied=218 ignored=0} inserts{seen=10850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:56.338301 27526 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Bootstrap complete.
I20250626 20:07:56.339391 27526 ts_tablet_manager.cc:1397] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Time spent bootstrapping tablet: real 0.056s user 0.040s sys 0.015s
I20250626 20:07:56.340270 27526 raft_consensus.cc:357] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:56.340482 27526 raft_consensus.cc:738] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa147f5bbf1479cb6393c93fc448716, State: Initialized, Role: FOLLOWER
I20250626 20:07:56.340574 27526 consensus_queue.cc:260] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:56.340770 27526 ts_tablet_manager.cc:1428] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20250626 20:07:56.340807 27634 heartbeater.cc:499] Master 127.20.190.254:32957 was elected leader, sending a full tablet report...
W20250626 20:07:56.378921 27638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 20:07:56.379077 27638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 20:07:56.379093 27638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 20:07:56.380244 27638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 20:07:56.380290 27638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.190.195
I20250626 20:07:56.381456 27638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.190.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.190.195:37755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.190.195
--webserver_port=39467
--tserver_master_addrs=127.20.190.254:32957
--never_fsync=true
--heap_profile_path=/tmp/kudu.27638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.190.195
--log_dir=/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type RELEASE
built by None at 26 Jun 2025 19:43:16 UTC on 5fd53c4cbb9d
build id 6783
I20250626 20:07:56.381637 27638 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 20:07:56.381826 27638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 20:07:56.384852 27647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.384887 27645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 20:07:56.384861 27644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 20:07:56.384953 27638 server_base.cc:1048] running on GCE node
I20250626 20:07:56.385262 27638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 20:07:56.385448 27638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 20:07:56.386572 27638 hybrid_clock.cc:648] HybridClock initialized: now 1750968476386556 us; error 36 us; skew 500 ppm
I20250626 20:07:56.387493 27638 webserver.cc:469] Webserver started at http://127.20.190.195:39467/ using document root <none> and password file <none>
I20250626 20:07:56.387650 27638 fs_manager.cc:362] Metadata directory not provided
I20250626 20:07:56.387696 27638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 20:07:56.388787 27638 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:56.389330 27653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.389488 27638 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:56.389540 27638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb"
format_stamp: "Formatted at 2025-06-26 20:07:55 on dist-test-slave-l47t"
I20250626 20:07:56.389722 27638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 20:07:56.401623 27638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 20:07:56.401820 27638 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 20:07:56.401895 27638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 20:07:56.402050 27638 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 20:07:56.402328 27660 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250626 20:07:56.403060 27638 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 20:07:56.403105 27638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:56.403126 27638 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 20:07:56.403605 27638 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 20:07:56.403636 27638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250626 20:07:56.403716 27660 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Bootstrap starting.
I20250626 20:07:56.409063 27638 rpc_server.cc:307] RPC server started. Bound to: 127.20.190.195:37755
I20250626 20:07:56.409127 27767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.190.195:37755 every 8 connection(s)
I20250626 20:07:56.409358 27638 server_base.cc:1180] Dumped server information to /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb
I20250626 20:07:56.413857 27768 heartbeater.cc:344] Connected to a master server at 127.20.190.254:32957
I20250626 20:07:56.413975 27768 heartbeater.cc:461] Registering TS with master...
I20250626 20:07:56.414192 27768 heartbeater.cc:507] Master 127.20.190.254:32957 requested a full tablet report, sending...
I20250626 20:07:56.414367 21243 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasksd6vjS/build/release/bin/kudu as pid 27638
I20250626 20:07:56.414739 27312 ts_manager.cc:194] Registered new tserver with Master: 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755)
I20250626 20:07:56.415089 27312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.190.195:39081
I20250626 20:07:56.417495 27660 log.cc:826] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Log is configured to *not* fsync() on all Append() calls
I20250626 20:07:56.418450 21243 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 20:07:56.447147 27660 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=218 overwritten=0 applied=218 ignored=0} inserts{seen=10850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 20:07:56.447423 27660 tablet_bootstrap.cc:492] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Bootstrap complete.
I20250626 20:07:56.448266 27660 ts_tablet_manager.cc:1397] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Time spent bootstrapping tablet: real 0.045s user 0.039s sys 0.004s
I20250626 20:07:56.449049 27660 raft_consensus.cc:357] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:56.449234 27660 raft_consensus.cc:738] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906f1c6e03ea4fe38483fa3f6fdb44fb, State: Initialized, Role: FOLLOWER
I20250626 20:07:56.449335 27660 consensus_queue.cc:260] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:56.449553 27660 ts_tablet_manager.cc:1428] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250626 20:07:56.449620 27768 heartbeater.cc:499] Master 127.20.190.254:32957 was elected leader, sending a full tablet report...
I20250626 20:07:57.807827 27779 raft_consensus.cc:491] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:57.807955 27779 raft_consensus.cc:513] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:57.808271 27779 leader_election.cc:290] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415), d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071)
I20250626 20:07:57.811566 27588 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 218 } } ignore_live_leader: false dest_uuid: "3fa147f5bbf1479cb6393c93fc448716" is_pre_election: true
I20250626 20:07:57.811576 27453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 218 } } ignore_live_leader: false dest_uuid: "d40be74472824d04ba807bfab764b6e4" is_pre_election: true
I20250626 20:07:57.811693 27453 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 1.
I20250626 20:07:57.811699 27588 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 1.
I20250626 20:07:57.811955 27654 leader_election.cc:304] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906f1c6e03ea4fe38483fa3f6fdb44fb, d40be74472824d04ba807bfab764b6e4; no voters:
I20250626 20:07:57.812075 27779 raft_consensus.cc:2802] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 20:07:57.812147 27779 raft_consensus.cc:491] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:57.812178 27779 raft_consensus.cc:3058] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:57.813035 27779 raft_consensus.cc:513] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:57.813135 27779 leader_election.cc:290] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 2 election: Requested vote from peers 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415), d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071)
I20250626 20:07:57.813308 27588 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 218 } } ignore_live_leader: false dest_uuid: "3fa147f5bbf1479cb6393c93fc448716"
I20250626 20:07:57.813308 27453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "80cdb32ba31540d1997f0990a3584aae" candidate_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 218 } } ignore_live_leader: false dest_uuid: "d40be74472824d04ba807bfab764b6e4"
I20250626 20:07:57.813378 27588 raft_consensus.cc:3058] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:57.813378 27453 raft_consensus.cc:3058] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 20:07:57.814392 27588 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 2.
I20250626 20:07:57.814392 27453 raft_consensus.cc:2466] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906f1c6e03ea4fe38483fa3f6fdb44fb in term 2.
I20250626 20:07:57.814579 27654 leader_election.cc:304] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906f1c6e03ea4fe38483fa3f6fdb44fb, d40be74472824d04ba807bfab764b6e4; no voters:
I20250626 20:07:57.814677 27779 raft_consensus.cc:2802] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 FOLLOWER]: Leader election won for term 2
I20250626 20:07:57.814772 27779 raft_consensus.cc:695] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 LEADER]: Becoming Leader. State: Replica: 906f1c6e03ea4fe38483fa3f6fdb44fb, State: Running, Role: LEADER
I20250626 20:07:57.814855 27779 consensus_queue.cc:237] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:57.815455 27312 catalog_manager.cc:5582] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb reported cstate change: term changed from 0 to 2, leader changed from <none> to 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195), VOTER 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194) added, VOTER 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195) added, VOTER d40be74472824d04ba807bfab764b6e4 (127.20.190.193) added. New cstate: current_term: 2 leader_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } health_report { overall_health: HEALTHY } } }
W20250626 20:07:57.937254 21243 scanner-internal.cc:458] Time spent opening tablet: real 1.510s user 0.001s sys 0.000s
I20250626 20:07:57.968461 27702 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:57.969408 27433 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:57.969671 27568 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
453db716f51a4ec780dd8187c8dbf2e5 | 127.20.190.254:32957 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3fa147f5bbf1479cb6393c93fc448716 | 127.20.190.194:37415 | HEALTHY | <none> | 0 | 0
906f1c6e03ea4fe38483fa3f6fdb44fb | 127.20.190.195:37755 | HEALTHY | <none> | 1 | 0
d40be74472824d04ba807bfab764b6e4 | 127.20.190.193:42071 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.190.193 | experimental | 127.20.190.193:42071
local_ip_for_outbound_sockets | 127.20.190.194 | experimental | 127.20.190.194:37415
local_ip_for_outbound_sockets | 127.20.190.195 | experimental | 127.20.190.195:37755
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.190.193:42071
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.190.194:37415
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.190.195:37755
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 80cdb32ba31540d1997f0990a3584aae of table 'pre_rebuild' is conflicted: 2 replicas' active configs disagree with the leader master's
3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415): RUNNING
d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071): RUNNING
906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755): RUNNING [LEADER]
All reported replicas are:
A = 3fa147f5bbf1479cb6393c93fc448716
B = d40be74472824d04ba807bfab764b6e4
C = 906f1c6e03ea4fe38483fa3f6fdb44fb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 2 | -1 | Yes
B | A B C | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+--------------------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250626 20:07:57.981334 21243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250626 20:07:58.151437 27568 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:58.151999 27433 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:58.152487 27702 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
453db716f51a4ec780dd8187c8dbf2e5 | 127.20.190.254:32957 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3fa147f5bbf1479cb6393c93fc448716 | 127.20.190.194:37415 | HEALTHY | <none> | 0 | 0
906f1c6e03ea4fe38483fa3f6fdb44fb | 127.20.190.195:37755 | HEALTHY | <none> | 1 | 0
d40be74472824d04ba807bfab764b6e4 | 127.20.190.193:42071 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.190.193 | experimental | 127.20.190.193:42071
local_ip_for_outbound_sockets | 127.20.190.194 | experimental | 127.20.190.194:37415
local_ip_for_outbound_sockets | 127.20.190.195 | experimental | 127.20.190.195:37755
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.190.193:42071
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.190.194:37415
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.190.195:37755
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 80cdb32ba31540d1997f0990a3584aae of table 'pre_rebuild' is conflicted: 2 replicas' active configs disagree with the leader master's
3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415): RUNNING
d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071): RUNNING
906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755): RUNNING [LEADER]
All reported replicas are:
A = 3fa147f5bbf1479cb6393c93fc448716
B = d40be74472824d04ba807bfab764b6e4
C = 906f1c6e03ea4fe38483fa3f6fdb44fb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 2 | -1 | Yes
B | A B C | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+--------------------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250626 20:07:58.162701 21243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250626 20:07:58.190716 27453 raft_consensus.cc:1273] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 2 FOLLOWER]: Refusing update from remote peer 906f1c6e03ea4fe38483fa3f6fdb44fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20250626 20:07:58.190971 27779 consensus_queue.cc:1035] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20250626 20:07:58.192231 27588 raft_consensus.cc:1273] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 2 FOLLOWER]: Refusing update from remote peer 906f1c6e03ea4fe38483fa3f6fdb44fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20250626 20:07:58.192435 27843 consensus_queue.cc:1035] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20250626 20:07:58.195142 27722 consensus_queue.cc:237] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:58.195509 27453 raft_consensus.cc:1273] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 2 FOLLOWER]: Refusing update from remote peer 906f1c6e03ea4fe38483fa3f6fdb44fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20250626 20:07:58.195678 27843 consensus_queue.cc:1035] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20250626 20:07:58.196141 27784 raft_consensus.cc:2953] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index -1 to 220, VOTER 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } }
I20250626 20:07:58.196269 27453 raft_consensus.cc:2953] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index -1 to 220, VOTER 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } }
I20250626 20:07:58.196928 27297 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 80cdb32ba31540d1997f0990a3584aae with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 20:07:58.197093 27312 catalog_manager.cc:5582] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb reported cstate change: config changed from index -1 to 220, VOTER 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194) evicted. New cstate: current_term: 2 leader_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:58.197952 27722 consensus_queue.cc:237] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 218, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } }
I20250626 20:07:58.198145 27784 raft_consensus.cc:2953] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 220 to 221, VOTER d40be74472824d04ba807bfab764b6e4 (127.20.190.193) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } }
I20250626 20:07:58.199030 27297 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 80cdb32ba31540d1997f0990a3584aae with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 20:07:58.199199 27312 catalog_manager.cc:5582] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb reported cstate change: config changed from index 220 to 221, VOTER d40be74472824d04ba807bfab764b6e4 (127.20.190.193) evicted. New cstate: current_term: 2 leader_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:58.200374 27568 tablet_service.cc:1515] Processing DeleteTablet for tablet 80cdb32ba31540d1997f0990a3584aae with delete_type TABLET_DATA_TOMBSTONED (TS 3fa147f5bbf1479cb6393c93fc448716 not found in new config with opid_index 220) from {username='slave'} at 127.0.0.1:35440
I20250626 20:07:58.200601 27851 tablet_replica.cc:331] stopping tablet replica
I20250626 20:07:58.200690 27851 raft_consensus.cc:2241] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 20:07:58.200745 27851 raft_consensus.cc:2270] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 20:07:58.201025 27851 ts_tablet_manager.cc:1905] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 20:07:58.201864 27433 tablet_service.cc:1515] Processing DeleteTablet for tablet 80cdb32ba31540d1997f0990a3584aae with delete_type TABLET_DATA_TOMBSTONED (TS d40be74472824d04ba807bfab764b6e4 not found in new config with opid_index 221) from {username='slave'} at 127.0.0.1:52394
I20250626 20:07:58.202060 27852 tablet_replica.cc:331] stopping tablet replica
I20250626 20:07:58.202122 27852 raft_consensus.cc:2241] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 20:07:58.202178 27852 raft_consensus.cc:2270] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 20:07:58.202276 27851 ts_tablet_manager.cc:1918] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.219
I20250626 20:07:58.202333 27851 log.cc:1199] T 80cdb32ba31540d1997f0990a3584aae P 3fa147f5bbf1479cb6393c93fc448716: Deleting WAL directory at /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/wal/wals/80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.202455 27852 ts_tablet_manager.cc:1905] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 20:07:58.202781 27296 catalog_manager.cc:4928] TS 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415): tablet 80cdb32ba31540d1997f0990a3584aae (table pre_rebuild [id=9fb8f5a3475048f59fccbae53d653d3a]) successfully deleted
I20250626 20:07:58.203627 27852 ts_tablet_manager.cc:1918] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.220
I20250626 20:07:58.203672 27852 log.cc:1199] T 80cdb32ba31540d1997f0990a3584aae P d40be74472824d04ba807bfab764b6e4: Deleting WAL directory at /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/wal/wals/80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.204017 27296 catalog_manager.cc:4928] TS d40be74472824d04ba807bfab764b6e4 (127.20.190.193:42071): tablet 80cdb32ba31540d1997f0990a3584aae (table pre_rebuild [id=9fb8f5a3475048f59fccbae53d653d3a]) successfully deleted
I20250626 20:07:58.408322 27702 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 20:07:58.408461 27568 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:58.409258 27433 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
453db716f51a4ec780dd8187c8dbf2e5 | 127.20.190.254:32957 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3fa147f5bbf1479cb6393c93fc448716 | 127.20.190.194:37415 | HEALTHY | <none> | 0 | 0
906f1c6e03ea4fe38483fa3f6fdb44fb | 127.20.190.195:37755 | HEALTHY | <none> | 1 | 0
d40be74472824d04ba807bfab764b6e4 | 127.20.190.193:42071 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.190.193 | experimental | 127.20.190.193:42071
local_ip_for_outbound_sockets | 127.20.190.194 | experimental | 127.20.190.194:37415
local_ip_for_outbound_sockets | 127.20.190.195 | experimental | 127.20.190.195:37755
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.190.193:42071
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.190.194:37415
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.190.195:37755
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 20:07:58.417197 21243 log_verifier.cc:126] Checking tablet 80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.429085 21243 log_verifier.cc:177] Verified matching terms for 221 ops in tablet 80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.429466 27311 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55552:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 20:07:58.429541 27311 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55552:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 20:07:58.430190 27311 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 453db716f51a4ec780dd8187c8dbf2e5: Sending DeleteTablet for 1 replicas of tablet 80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.430456 21243 test_util.cc:276] Using random seed: 506164470
I20250626 20:07:58.430460 27702 tablet_service.cc:1515] Processing DeleteTablet for tablet 80cdb32ba31540d1997f0990a3584aae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 20:07:58 UTC) from {username='slave'} at 127.0.0.1:37226
I20250626 20:07:58.430651 27875 tablet_replica.cc:331] stopping tablet replica
I20250626 20:07:58.430730 27875 raft_consensus.cc:2241] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 LEADER]: Raft consensus shutting down.
I20250626 20:07:58.430788 27875 raft_consensus.cc:2270] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 20:07:58.431836 27875 ts_tablet_manager.cc:1905] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 20:07:58.433445 27875 ts_tablet_manager.cc:1918] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.221
I20250626 20:07:58.433507 27875 log.cc:1199] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Deleting WAL directory at /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/wal/wals/80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.433804 27875 ts_tablet_manager.cc:1939] T 80cdb32ba31540d1997f0990a3584aae P 906f1c6e03ea4fe38483fa3f6fdb44fb: Deleting consensus metadata
I20250626 20:07:58.434160 27297 catalog_manager.cc:4928] TS 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755): tablet 80cdb32ba31540d1997f0990a3584aae (table pre_rebuild [id=9fb8f5a3475048f59fccbae53d653d3a]) successfully deleted
I20250626 20:07:58.434820 27311 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41776:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 20:07:58.435068 27311 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 20:07:58.436378 27702 tablet_service.cc:1468] Processing CreateTablet for tablet 5e5d7576fdb04f03b9770af02f8464b7 (DEFAULT_TABLE table=post_rebuild [id=bd6d37e77bb84ce3b0316e209c86e253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:58.436406 27568 tablet_service.cc:1468] Processing CreateTablet for tablet 5e5d7576fdb04f03b9770af02f8464b7 (DEFAULT_TABLE table=post_rebuild [id=bd6d37e77bb84ce3b0316e209c86e253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:58.436411 27433 tablet_service.cc:1468] Processing CreateTablet for tablet 5e5d7576fdb04f03b9770af02f8464b7 (DEFAULT_TABLE table=post_rebuild [id=bd6d37e77bb84ce3b0316e209c86e253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 20:07:58.436518 27702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e5d7576fdb04f03b9770af02f8464b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:58.436527 27433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e5d7576fdb04f03b9770af02f8464b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:58.436540 27568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e5d7576fdb04f03b9770af02f8464b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 20:07:58.437371 27881 tablet_bootstrap.cc:492] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716: Bootstrap starting.
I20250626 20:07:58.437706 27881 tablet_bootstrap.cc:654] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:58.437709 27882 tablet_bootstrap.cc:492] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb: Bootstrap starting.
I20250626 20:07:58.437816 27883 tablet_bootstrap.cc:492] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4: Bootstrap starting.
I20250626 20:07:58.437971 27882 tablet_bootstrap.cc:654] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:58.438153 27883 tablet_bootstrap.cc:654] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4: Neither blocks nor log segments found. Creating new log.
I20250626 20:07:58.438380 27881 tablet_bootstrap.cc:492] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716: No bootstrap required, opened a new log
I20250626 20:07:58.438432 27881 ts_tablet_manager.cc:1397] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:58.438568 27881 raft_consensus.cc:357] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.438634 27881 raft_consensus.cc:383] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:58.438668 27881 raft_consensus.cc:738] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa147f5bbf1479cb6393c93fc448716, State: Initialized, Role: FOLLOWER
I20250626 20:07:58.438669 27883 tablet_bootstrap.cc:492] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4: No bootstrap required, opened a new log
I20250626 20:07:58.438709 27883 ts_tablet_manager.cc:1397] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250626 20:07:58.438731 27881 consensus_queue.cc:260] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.438848 27881 ts_tablet_manager.cc:1428] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250626 20:07:58.438853 27883 raft_consensus.cc:357] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.438906 27883 raft_consensus.cc:383] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:58.438939 27883 raft_consensus.cc:738] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40be74472824d04ba807bfab764b6e4, State: Initialized, Role: FOLLOWER
I20250626 20:07:58.439000 27883 consensus_queue.cc:260] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.439091 27883 ts_tablet_manager.cc:1428] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:58.439358 27882 tablet_bootstrap.cc:492] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb: No bootstrap required, opened a new log
I20250626 20:07:58.439406 27882 ts_tablet_manager.cc:1397] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 20:07:58.439500 27882 raft_consensus.cc:357] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.439563 27882 raft_consensus.cc:383] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 20:07:58.439594 27882 raft_consensus.cc:738] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906f1c6e03ea4fe38483fa3f6fdb44fb, State: Initialized, Role: FOLLOWER
I20250626 20:07:58.439630 27882 consensus_queue.cc:260] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.439716 27882 ts_tablet_manager.cc:1428] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250626 20:07:58.484788 27840 raft_consensus.cc:491] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 20:07:58.484858 27840 raft_consensus.cc:513] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.485108 27840 leader_election.cc:290] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755), 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415)
I20250626 20:07:58.487428 27722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e5d7576fdb04f03b9770af02f8464b7" candidate_uuid: "d40be74472824d04ba807bfab764b6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" is_pre_election: true
I20250626 20:07:58.487461 27588 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e5d7576fdb04f03b9770af02f8464b7" candidate_uuid: "d40be74472824d04ba807bfab764b6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fa147f5bbf1479cb6393c93fc448716" is_pre_election: true
I20250626 20:07:58.487519 27588 raft_consensus.cc:2466] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40be74472824d04ba807bfab764b6e4 in term 0.
I20250626 20:07:58.487524 27722 raft_consensus.cc:2466] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d40be74472824d04ba807bfab764b6e4 in term 0.
I20250626 20:07:58.487694 27386 leader_election.cc:304] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906f1c6e03ea4fe38483fa3f6fdb44fb, d40be74472824d04ba807bfab764b6e4; no voters:
I20250626 20:07:58.487818 27840 raft_consensus.cc:2802] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 20:07:58.487877 27840 raft_consensus.cc:491] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 20:07:58.487892 27840 raft_consensus.cc:3058] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:58.488423 27840 raft_consensus.cc:513] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.488519 27840 leader_election.cc:290] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [CANDIDATE]: Term 1 election: Requested vote from peers 906f1c6e03ea4fe38483fa3f6fdb44fb (127.20.190.195:37755), 3fa147f5bbf1479cb6393c93fc448716 (127.20.190.194:37415)
I20250626 20:07:58.488692 27722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e5d7576fdb04f03b9770af02f8464b7" candidate_uuid: "d40be74472824d04ba807bfab764b6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb"
I20250626 20:07:58.488741 27588 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e5d7576fdb04f03b9770af02f8464b7" candidate_uuid: "d40be74472824d04ba807bfab764b6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fa147f5bbf1479cb6393c93fc448716"
I20250626 20:07:58.488770 27722 raft_consensus.cc:3058] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:58.488797 27588 raft_consensus.cc:3058] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 0 FOLLOWER]: Advancing to term 1
I20250626 20:07:58.489282 27722 raft_consensus.cc:2466] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40be74472824d04ba807bfab764b6e4 in term 1.
I20250626 20:07:58.489291 27588 raft_consensus.cc:2466] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d40be74472824d04ba807bfab764b6e4 in term 1.
I20250626 20:07:58.489441 27386 leader_election.cc:304] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906f1c6e03ea4fe38483fa3f6fdb44fb, d40be74472824d04ba807bfab764b6e4; no voters:
I20250626 20:07:58.489512 27840 raft_consensus.cc:2802] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 20:07:58.489630 27840 raft_consensus.cc:695] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [term 1 LEADER]: Becoming Leader. State: Replica: d40be74472824d04ba807bfab764b6e4, State: Running, Role: LEADER
I20250626 20:07:58.489688 27840 consensus_queue.cc:237] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } }
I20250626 20:07:58.490253 27312 catalog_manager.cc:5582] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d40be74472824d04ba807bfab764b6e4 (127.20.190.193). New cstate: current_term: 1 leader_uuid: "d40be74472824d04ba807bfab764b6e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40be74472824d04ba807bfab764b6e4" member_type: VOTER last_known_addr { host: "127.20.190.193" port: 42071 } health_report { overall_health: HEALTHY } } }
I20250626 20:07:58.504798 27588 raft_consensus.cc:1273] T 5e5d7576fdb04f03b9770af02f8464b7 P 3fa147f5bbf1479cb6393c93fc448716 [term 1 FOLLOWER]: Refusing update from remote peer d40be74472824d04ba807bfab764b6e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 20:07:58.505043 27840 consensus_queue.cc:1035] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fa147f5bbf1479cb6393c93fc448716" member_type: VOTER last_known_addr { host: "127.20.190.194" port: 37415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:58.505089 27722 raft_consensus.cc:1273] T 5e5d7576fdb04f03b9770af02f8464b7 P 906f1c6e03ea4fe38483fa3f6fdb44fb [term 1 FOLLOWER]: Refusing update from remote peer d40be74472824d04ba807bfab764b6e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 20:07:58.505267 27840 consensus_queue.cc:1035] T 5e5d7576fdb04f03b9770af02f8464b7 P d40be74472824d04ba807bfab764b6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "906f1c6e03ea4fe38483fa3f6fdb44fb" member_type: VOTER last_known_addr { host: "127.20.190.195" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 20:07:58.506592 27844 mvcc.cc:204] Tried to move back new op lower bound from 7171966887954137088 to 7171966887894224896. Current Snapshot: MvccSnapshot[applied={T|T < 7171966887954137088}]
I20250626 20:07:58.507203 27842 mvcc.cc:204] Tried to move back new op lower bound from 7171966887955718144 to 7171966887894224896. Current Snapshot: MvccSnapshot[applied={T|T < 7171966887954137088}]
I20250626 20:07:58.507321 27848 mvcc.cc:204] Tried to move back new op lower bound from 7171966887954137088 to 7171966887894224896. Current Snapshot: MvccSnapshot[applied={T|T < 7171966887954137088}]
I20250626 20:07:58.657315 27568 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:58.657310 27702 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 20:07:58.657970 27433 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
453db716f51a4ec780dd8187c8dbf2e5 | 127.20.190.254:32957 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3fa147f5bbf1479cb6393c93fc448716 | 127.20.190.194:37415 | HEALTHY | <none> | 0 | 0
906f1c6e03ea4fe38483fa3f6fdb44fb | 127.20.190.195:37755 | HEALTHY | <none> | 0 | 0
d40be74472824d04ba807bfab764b6e4 | 127.20.190.193:42071 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.190.193 | experimental | 127.20.190.193:42071
local_ip_for_outbound_sockets | 127.20.190.194 | experimental | 127.20.190.194:37415
local_ip_for_outbound_sockets | 127.20.190.195 | experimental | 127.20.190.195:37755
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.190.193:42071
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.190.194:37415
server_dump_info_path | /tmp/dist-test-tasksd6vjS/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750968436877382-21243-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.190.195:37755
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.20.190.212:42647 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 20:07:58.665508 21243 log_verifier.cc:126] Checking tablet 5e5d7576fdb04f03b9770af02f8464b7
I20250626 20:07:58.697266 21243 log_verifier.cc:177] Verified matching terms for 215 ops in tablet 5e5d7576fdb04f03b9770af02f8464b7
I20250626 20:07:58.697329 21243 log_verifier.cc:126] Checking tablet 80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.697378 21243 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 80cdb32ba31540d1997f0990a3584aae
I20250626 20:07:58.699074 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 27351
I20250626 20:07:58.704460 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 27504
I20250626 20:07:58.710384 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 27638
I20250626 20:07:58.714893 21243 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasksd6vjS/build/release/bin/kudu with pid 27281
2025-06-26T20:07:58Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (3925 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (3925 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (41853 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST