Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250624 08:02:13.646512 27543 test_util.cc:276] Using random seed: -1090254689
I20250624 08:02:13.652938 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:13.652992 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:13.653012 27543 ts_itest-base.cc:117] 4 tablet servers
I20250624 08:02:13.653026 27543 ts_itest-base.cc:118] 3 replicas per TS
I20250624 08:02:13.653036 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:13Z Disabled control of system clock
I20250624 08:02:13.661768 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:33439
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:33439 with env {}
W20250624 08:02:13.756773 27555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:13.756979 27555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:13.757022 27555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:13.760095 27555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:13.760145 27555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:13.760164 27555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:13.760181 27555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:13.763803 27555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:33439
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.27555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:13.764858 27555 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:13.765945 27555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:13.772396 27563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:13.772387 27561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:13.772531 27555 server_base.cc:1048] running on GCE node
W20250624 08:02:13.772421 27560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:13.773049 27555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:13.773886 27555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:13.775038 27555 hybrid_clock.cc:648] HybridClock initialized: now 1750752133775013 us; error 37 us; skew 500 ppm
I20250624 08:02:13.776909 27555 webserver.cc:469] Webserver started at http://127.26.229.254:35789/ using document root <none> and password file <none>
I20250624 08:02:13.777451 27555 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:13.777506 27555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:13.777699 27555 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:13.779328 27555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2660612a779642b99aaa5f0083c8eede"
format_stamp: "Formatted at 2025-06-24 08:02:13 on dist-test-slave-4j8z"
I20250624 08:02:13.779801 27555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2660612a779642b99aaa5f0083c8eede"
format_stamp: "Formatted at 2025-06-24 08:02:13 on dist-test-slave-4j8z"
I20250624 08:02:13.783043 27555 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 08:02:13.785081 27569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:13.785982 27555 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:13.786113 27555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2660612a779642b99aaa5f0083c8eede"
format_stamp: "Formatted at 2025-06-24 08:02:13 on dist-test-slave-4j8z"
I20250624 08:02:13.786192 27555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:13.814352 27555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:13.815088 27555 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:13.815284 27555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:13.821905 27555 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:33439
I20250624 08:02:13.821910 27621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:33439 every 8 connection(s)
I20250624 08:02:13.822893 27555 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:13.825275 27622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:13.827384 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 27555
I20250624 08:02:13.827555 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:13.831022 27622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Bootstrap starting.
I20250624 08:02:13.833474 27622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:13.834314 27622 log.cc:826] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:13.835973 27622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: No bootstrap required, opened a new log
I20250624 08:02:13.838411 27622 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:13.838614 27622 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:13.838740 27622 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660612a779642b99aaa5f0083c8eede, State: Initialized, Role: FOLLOWER
I20250624 08:02:13.839200 27622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:13.839349 27622 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:13.839421 27622 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:13.839547 27622 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:13.840382 27622 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:13.840678 27622 leader_election.cc:304] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2660612a779642b99aaa5f0083c8eede; no voters:
I20250624 08:02:13.840922 27622 leader_election.cc:290] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:13.841027 27627 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:13.841279 27627 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 LEADER]: Becoming Leader. State: Replica: 2660612a779642b99aaa5f0083c8eede, State: Running, Role: LEADER
I20250624 08:02:13.842033 27622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:13.842525 27627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:13.844632 27629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2660612a779642b99aaa5f0083c8eede. Latest consensus state: current_term: 1 leader_uuid: "2660612a779642b99aaa5f0083c8eede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } } }
I20250624 08:02:13.844738 27629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:13.845322 27628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2660612a779642b99aaa5f0083c8eede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } } }
I20250624 08:02:13.845404 27628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: This master's current role is: LEADER
W20250624 08:02:13.847204 27641 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250624 08:02:13.847301 27641 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250624 08:02:13.847373 27642 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:13.848147 27642 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:13.854933 27642 catalog_manager.cc:1349] Generated new cluster ID: f83a7b48ecf34f388f287593b2fe4e7f
I20250624 08:02:13.854997 27642 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:13.905405 27642 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:13.906714 27642 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:13.918340 27642 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Generated new TSK 0
I20250624 08:02:13.919260 27642 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:13.933066 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--builtin_ntp_servers=127.26.229.212:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 08:02:14.034133 27646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:14.034350 27646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:14.034420 27646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:14.037449 27646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:14.037580 27646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:14.040921 27646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--never_fsync=true
--heap_profile_path=/tmp/kudu.27646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:14.041922 27646 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:14.043080 27646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:14.050266 27652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.050282 27651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.050282 27654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:14.050482 27646 server_base.cc:1048] running on GCE node
I20250624 08:02:14.050837 27646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:14.051389 27646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:14.052554 27646 hybrid_clock.cc:648] HybridClock initialized: now 1750752134052529 us; error 36 us; skew 500 ppm
I20250624 08:02:14.054315 27646 webserver.cc:469] Webserver started at http://127.26.229.193:45335/ using document root <none> and password file <none>
I20250624 08:02:14.054886 27646 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:14.054944 27646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:14.055127 27646 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:14.056614 27646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a759884bd8ab40a281916d17c4e9b6ce"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.057072 27646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a759884bd8ab40a281916d17c4e9b6ce"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.060252 27646 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:14.062209 27660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.063160 27646 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:14.063268 27646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a759884bd8ab40a281916d17c4e9b6ce"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.063350 27646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:14.083055 27646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:14.083663 27646 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:14.083839 27646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:14.084347 27646 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:14.085229 27646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:14.085278 27646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.085336 27646 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:14.085371 27646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.094053 27646 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:36387
I20250624 08:02:14.094081 27773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:36387 every 8 connection(s)
I20250624 08:02:14.094995 27646 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:14.098778 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 27646
I20250624 08:02:14.098913 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:14.100867 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--builtin_ntp_servers=127.26.229.212:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:14.103962 27774 heartbeater.cc:344] Connected to a master server at 127.26.229.254:33439
I20250624 08:02:14.104242 27774 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:14.104903 27774 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:14.106530 27586 ts_manager.cc:194] Registered new tserver with Master: a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193:36387)
I20250624 08:02:14.108295 27586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:54135
W20250624 08:02:14.201572 27777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:14.201761 27777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:14.201802 27777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:14.204854 27777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:14.204946 27777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:14.208367 27777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--never_fsync=true
--heap_profile_path=/tmp/kudu.27777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:14.209256 27777 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:14.210305 27777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:14.217710 27782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.217727 27785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.217748 27783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:14.217911 27777 server_base.cc:1048] running on GCE node
I20250624 08:02:14.218354 27777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:14.218889 27777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:14.220029 27777 hybrid_clock.cc:648] HybridClock initialized: now 1750752134220014 us; error 34 us; skew 500 ppm
I20250624 08:02:14.221637 27777 webserver.cc:469] Webserver started at http://127.26.229.194:37121/ using document root <none> and password file <none>
I20250624 08:02:14.222148 27777 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:14.222208 27777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:14.222375 27777 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:14.223863 27777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "eb5ee161f5504194823dd408d02f865a"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.224282 27777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "eb5ee161f5504194823dd408d02f865a"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.227437 27777 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 08:02:14.229288 27791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.230161 27777 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:14.230266 27777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eb5ee161f5504194823dd408d02f865a"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.230341 27777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:14.249063 27777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:14.249606 27777 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:14.249758 27777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:14.250222 27777 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:14.251178 27777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:14.251232 27777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.251304 27777 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:14.251338 27777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.260056 27777 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:33395
I20250624 08:02:14.260079 27904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:33395 every 8 connection(s)
I20250624 08:02:14.260919 27777 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:14.265945 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 27777
I20250624 08:02:14.266026 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 08:02:14.268065 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--builtin_ntp_servers=127.26.229.212:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:14.269578 27905 heartbeater.cc:344] Connected to a master server at 127.26.229.254:33439
I20250624 08:02:14.269786 27905 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:14.270244 27905 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:14.271257 27586 ts_manager.cc:194] Registered new tserver with Master: eb5ee161f5504194823dd408d02f865a (127.26.229.194:33395)
I20250624 08:02:14.272082 27586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:55329
W20250624 08:02:14.365432 27908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:14.365620 27908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:14.365659 27908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:14.368693 27908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:14.368782 27908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:14.372242 27908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--never_fsync=true
--heap_profile_path=/tmp/kudu.27908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:14.373138 27908 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:14.374087 27908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:14.380918 27913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.380925 27914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.380935 27916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:14.381153 27908 server_base.cc:1048] running on GCE node
I20250624 08:02:14.381481 27908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:14.381944 27908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:14.383078 27908 hybrid_clock.cc:648] HybridClock initialized: now 1750752134383060 us; error 32 us; skew 500 ppm
I20250624 08:02:14.384680 27908 webserver.cc:469] Webserver started at http://127.26.229.195:36411/ using document root <none> and password file <none>
I20250624 08:02:14.385167 27908 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:14.385219 27908 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:14.385391 27908 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:14.386808 27908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a8f3360ae7f0436686505c1b7616c330"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.387245 27908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a8f3360ae7f0436686505c1b7616c330"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.390197 27908 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:14.392184 27922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.393086 27908 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250624 08:02:14.393172 27908 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a8f3360ae7f0436686505c1b7616c330"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.393254 27908 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:14.407492 27908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:14.408010 27908 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:14.408177 27908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:14.408684 27908 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:14.409536 27908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:14.409579 27908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.409621 27908 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:14.409654 27908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.418335 27908 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:44345
I20250624 08:02:14.418371 28035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:44345 every 8 connection(s)
I20250624 08:02:14.419246 27908 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:14.422986 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 27908
I20250624 08:02:14.423061 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 08:02:14.424963 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.196:0
--local_ip_for_outbound_sockets=127.26.229.196
--webserver_interface=127.26.229.196
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--builtin_ntp_servers=127.26.229.212:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:14.427875 28036 heartbeater.cc:344] Connected to a master server at 127.26.229.254:33439
I20250624 08:02:14.428109 28036 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:14.428591 28036 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:14.429553 27586 ts_manager.cc:194] Registered new tserver with Master: a8f3360ae7f0436686505c1b7616c330 (127.26.229.195:44345)
I20250624 08:02:14.430348 27586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:50837
W20250624 08:02:14.526280 28039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:14.526499 28039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:14.526543 28039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:14.529615 28039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:14.529727 28039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.196
I20250624 08:02:14.533397 28039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.26.229.196
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33439
--never_fsync=true
--heap_profile_path=/tmp/kudu.28039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.196
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:14.534466 28039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:14.535605 28039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:14.543043 28045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.543051 28044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:14.543066 28047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:14.543234 28039 server_base.cc:1048] running on GCE node
I20250624 08:02:14.543676 28039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:14.544219 28039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:14.545367 28039 hybrid_clock.cc:648] HybridClock initialized: now 1750752134545354 us; error 26 us; skew 500 ppm
I20250624 08:02:14.547145 28039 webserver.cc:469] Webserver started at http://127.26.229.196:46153/ using document root <none> and password file <none>
I20250624 08:02:14.547688 28039 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:14.547755 28039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:14.547931 28039 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:14.549427 28039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d73579fa2de84bda98141d07ff1c9b94"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.549889 28039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d73579fa2de84bda98141d07ff1c9b94"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.553156 28039 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:14.555213 28053 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.556205 28039 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:14.556317 28039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d73579fa2de84bda98141d07ff1c9b94"
format_stamp: "Formatted at 2025-06-24 08:02:14 on dist-test-slave-4j8z"
I20250624 08:02:14.556399 28039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:14.589414 28039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:14.590143 28039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:14.590315 28039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:14.590893 28039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:14.591821 28039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:14.591872 28039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.591936 28039 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:14.591964 28039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:14.601063 28039 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.196:35095
I20250624 08:02:14.601097 28166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.196:35095 every 8 connection(s)
I20250624 08:02:14.602044 28039 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250624 08:02:14.610471 28167 heartbeater.cc:344] Connected to a master server at 127.26.229.254:33439
I20250624 08:02:14.610702 28167 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:14.611356 28167 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:14.611379 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28039
I20250624 08:02:14.611476 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250624 08:02:14.612442 27586 ts_manager.cc:194] Registered new tserver with Master: d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196:35095)
I20250624 08:02:14.613066 27586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.196:55145
I20250624 08:02:14.614174 27543 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250624 08:02:14.625778 27585 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34762:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 08:02:14.643476 28101 tablet_service.cc:1468] Processing CreateTablet for tablet 178abd11d618475d913d40de74d98c1a (DEFAULT_TABLE table=TestTable [id=7902cc7b8f884d8db5c13db68b3ac533]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:14.643474 27708 tablet_service.cc:1468] Processing CreateTablet for tablet 178abd11d618475d913d40de74d98c1a (DEFAULT_TABLE table=TestTable [id=7902cc7b8f884d8db5c13db68b3ac533]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:14.643474 27970 tablet_service.cc:1468] Processing CreateTablet for tablet 178abd11d618475d913d40de74d98c1a (DEFAULT_TABLE table=TestTable [id=7902cc7b8f884d8db5c13db68b3ac533]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:14.644378 28101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178abd11d618475d913d40de74d98c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:14.644405 27970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178abd11d618475d913d40de74d98c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:14.644405 27708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178abd11d618475d913d40de74d98c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:14.649102 28184 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Bootstrap starting.
I20250624 08:02:14.649839 28185 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: Bootstrap starting.
I20250624 08:02:14.650563 28186 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: Bootstrap starting.
I20250624 08:02:14.651449 28184 tablet_bootstrap.cc:654] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:14.652041 28185 tablet_bootstrap.cc:654] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:14.652329 28184 log.cc:826] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:14.652498 28186 tablet_bootstrap.cc:654] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:14.652801 28185 log.cc:826] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:14.653194 28186 log.cc:826] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:14.654002 28184 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: No bootstrap required, opened a new log
I20250624 08:02:14.654186 28184 ts_tablet_manager.cc:1397] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250624 08:02:14.654325 28185 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: No bootstrap required, opened a new log
I20250624 08:02:14.654501 28185 ts_tablet_manager.cc:1397] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250624 08:02:14.654628 28186 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: No bootstrap required, opened a new log
I20250624 08:02:14.654826 28186 ts_tablet_manager.cc:1397] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250624 08:02:14.656819 28186 raft_consensus.cc:357] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.657125 28186 raft_consensus.cc:383] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:14.657051 28184 raft_consensus.cc:357] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.657100 28185 raft_consensus.cc:357] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.657254 28184 raft_consensus.cc:383] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:14.657254 28185 raft_consensus.cc:383] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:14.657307 28184 raft_consensus.cc:738] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73579fa2de84bda98141d07ff1c9b94, State: Initialized, Role: FOLLOWER
I20250624 08:02:14.657307 28185 raft_consensus.cc:738] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a759884bd8ab40a281916d17c4e9b6ce, State: Initialized, Role: FOLLOWER
I20250624 08:02:14.657325 28186 raft_consensus.cc:738] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f3360ae7f0436686505c1b7616c330, State: Initialized, Role: FOLLOWER
I20250624 08:02:14.657724 28185 consensus_queue.cc:260] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.657723 28184 consensus_queue.cc:260] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.657863 28186 consensus_queue.cc:260] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.658432 28036 heartbeater.cc:499] Master 127.26.229.254:33439 was elected leader, sending a full tablet report...
I20250624 08:02:14.658524 28184 ts_tablet_manager.cc:1428] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:14.658524 28185 ts_tablet_manager.cc:1428] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:14.658947 27774 heartbeater.cc:499] Master 127.26.229.254:33439 was elected leader, sending a full tablet report...
I20250624 08:02:14.659626 28186 ts_tablet_manager.cc:1428] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250624 08:02:14.659816 28167 heartbeater.cc:499] Master 127.26.229.254:33439 was elected leader, sending a full tablet report...
W20250624 08:02:14.670068 28037 tablet.cc:2378] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:14.698326 28192 raft_consensus.cc:491] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:14.698526 28192 raft_consensus.cc:513] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.699847 28192 leader_election.cc:290] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193:36387), d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196:35095)
I20250624 08:02:14.703197 27728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "a8f3360ae7f0436686505c1b7616c330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a759884bd8ab40a281916d17c4e9b6ce" is_pre_election: true
I20250624 08:02:14.703462 27728 raft_consensus.cc:2466] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f3360ae7f0436686505c1b7616c330 in term 0.
I20250624 08:02:14.703536 28121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "a8f3360ae7f0436686505c1b7616c330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73579fa2de84bda98141d07ff1c9b94" is_pre_election: true
I20250624 08:02:14.703778 28121 raft_consensus.cc:2466] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f3360ae7f0436686505c1b7616c330 in term 0.
I20250624 08:02:14.703924 27925 leader_election.cc:304] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a759884bd8ab40a281916d17c4e9b6ce, a8f3360ae7f0436686505c1b7616c330; no voters:
I20250624 08:02:14.704154 28192 raft_consensus.cc:2802] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:14.704228 28192 raft_consensus.cc:491] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:14.704293 28192 raft_consensus.cc:3058] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:14.705140 28192 raft_consensus.cc:513] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.705466 28192 leader_election.cc:290] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [CANDIDATE]: Term 1 election: Requested vote from peers a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193:36387), d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196:35095)
I20250624 08:02:14.705830 28121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "a8f3360ae7f0436686505c1b7616c330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73579fa2de84bda98141d07ff1c9b94"
I20250624 08:02:14.705801 27728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "a8f3360ae7f0436686505c1b7616c330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a759884bd8ab40a281916d17c4e9b6ce"
I20250624 08:02:14.705953 28121 raft_consensus.cc:3058] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:14.705953 27728 raft_consensus.cc:3058] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:14.707152 27728 raft_consensus.cc:2466] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f3360ae7f0436686505c1b7616c330 in term 1.
I20250624 08:02:14.707439 27925 leader_election.cc:304] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a759884bd8ab40a281916d17c4e9b6ce, a8f3360ae7f0436686505c1b7616c330; no voters:
I20250624 08:02:14.707530 28121 raft_consensus.cc:2466] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f3360ae7f0436686505c1b7616c330 in term 1.
I20250624 08:02:14.707664 28192 raft_consensus.cc:2802] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:14.707906 28192 raft_consensus.cc:695] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [term 1 LEADER]: Becoming Leader. State: Replica: a8f3360ae7f0436686505c1b7616c330, State: Running, Role: LEADER
I20250624 08:02:14.708230 28192 consensus_queue.cc:237] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:14.711230 27586 catalog_manager.cc:5582] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8f3360ae7f0436686505c1b7616c330 (127.26.229.195). New cstate: current_term: 1 leader_uuid: "a8f3360ae7f0436686505c1b7616c330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:14.734851 27543 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250624 08:02:14.736471 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a759884bd8ab40a281916d17c4e9b6ce to finish bootstrapping
I20250624 08:02:14.742030 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a8f3360ae7f0436686505c1b7616c330 to finish bootstrapping
I20250624 08:02:14.746642 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d73579fa2de84bda98141d07ff1c9b94 to finish bootstrapping
I20250624 08:02:14.750923 27543 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250624 08:02:14.752069 27543 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "178abd11d618475d913d40de74d98c1a"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce"
rpc_addresses {
host: "127.26.229.193"
port: 36387
}
}
ts_infos {
permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94"
rpc_addresses {
host: "127.26.229.196"
port: 35095
}
}
ts_infos {
permanent_uuid: "a8f3360ae7f0436686505c1b7616c330"
rpc_addresses {
host: "127.26.229.195"
port: 44345
}
}
W20250624 08:02:14.846410 27775 tablet.cc:2378] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 08:02:14.852907 28168 tablet.cc:2378] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:15.114423 28192 consensus_queue.cc:1035] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:15.123008 28194 consensus_queue.cc:1035] T 178abd11d618475d913d40de74d98c1a P a8f3360ae7f0436686505c1b7616c330 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:15.125860 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 27908
W20250624 08:02:15.131839 28055 connection.cc:537] server connection from 127.26.229.195:44031 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 08:02:15.132454 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 27555
I20250624 08:02:15.141559 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:33439
--webserver_interface=127.26.229.254
--webserver_port=35789
--builtin_ntp_servers=127.26.229.212:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:33439 with env {}
W20250624 08:02:15.243284 28203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:15.243484 28203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:15.243551 28203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:15.246461 28203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:15.246515 28203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:15.246548 28203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:15.246578 28203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:15.250649 28203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:33439
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=35789
--never_fsync=true
--heap_profile_path=/tmp/kudu.28203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:15.251765 28203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:15.252836 28203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:15.259079 28209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:15.259136 28211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:15.259205 28208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:15.259408 28203 server_base.cc:1048] running on GCE node
I20250624 08:02:15.259831 28203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:15.260670 28203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:15.261818 28203 hybrid_clock.cc:648] HybridClock initialized: now 1750752135261794 us; error 40 us; skew 500 ppm
I20250624 08:02:15.263556 28203 webserver.cc:469] Webserver started at http://127.26.229.254:35789/ using document root <none> and password file <none>
I20250624 08:02:15.264104 28203 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:15.264181 28203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:15.267477 28203 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20250624 08:02:15.269237 28217 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:15.270108 28203 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:15.270206 28203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2660612a779642b99aaa5f0083c8eede"
format_stamp: "Formatted at 2025-06-24 08:02:13 on dist-test-slave-4j8z"
I20250624 08:02:15.270581 28203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:15.273998 27905 heartbeater.cc:499] Master 127.26.229.254:33439 was elected leader, sending a full tablet report...
W20250624 08:02:15.274891 27905 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:33439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:33439: connect: Connection refused (error 111)
I20250624 08:02:15.281787 28203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:15.282352 28203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:15.282531 28203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:15.289414 28203 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:33439
I20250624 08:02:15.289423 28270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:33439 every 8 connection(s)
I20250624 08:02:15.290463 28203 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:15.294106 28271 sys_catalog.cc:263] Verifying existing consensus state
I20250624 08:02:15.295917 28271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Bootstrap starting.
I20250624 08:02:15.296758 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28203
I20250624 08:02:15.296895 27543 kudu-admin-test.cc:735] Forcing unsafe config change on tserver d73579fa2de84bda98141d07ff1c9b94
I20250624 08:02:15.308764 28271 log.cc:826] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:15.315665 28271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:15.315986 28271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Bootstrap complete.
I20250624 08:02:15.319293 28271 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:15.319584 28271 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660612a779642b99aaa5f0083c8eede, State: Initialized, Role: FOLLOWER
I20250624 08:02:15.320058 28271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:15.320170 28271 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:15.320230 28271 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:15.320326 28271 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:15.321671 28271 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:15.321969 28271 leader_election.cc:304] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2660612a779642b99aaa5f0083c8eede; no voters:
I20250624 08:02:15.322255 28271 leader_election.cc:290] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 08:02:15.322348 28275 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:15.322531 28275 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [term 2 LEADER]: Becoming Leader. State: Replica: 2660612a779642b99aaa5f0083c8eede, State: Running, Role: LEADER
I20250624 08:02:15.322813 28275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } }
I20250624 08:02:15.323314 28271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:15.323602 28276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2660612a779642b99aaa5f0083c8eede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } } }
I20250624 08:02:15.323709 28276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:15.323652 28277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2660612a779642b99aaa5f0083c8eede. Latest consensus state: current_term: 2 leader_uuid: "2660612a779642b99aaa5f0083c8eede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660612a779642b99aaa5f0083c8eede" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33439 } } }
I20250624 08:02:15.323776 28277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:15.324025 28279 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:15.327349 28279 catalog_manager.cc:671] Loaded metadata for table TestTable [id=7902cc7b8f884d8db5c13db68b3ac533]
I20250624 08:02:15.328511 28279 tablet_loader.cc:96] loaded metadata for tablet 178abd11d618475d913d40de74d98c1a (table TestTable [id=7902cc7b8f884d8db5c13db68b3ac533])
I20250624 08:02:15.328653 28279 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:15.329260 28279 catalog_manager.cc:1261] Loaded cluster ID: f83a7b48ecf34f388f287593b2fe4e7f
I20250624 08:02:15.329319 28279 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:15.331722 28279 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:15.332412 28279 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2660612a779642b99aaa5f0083c8eede: Loaded TSK: 0
I20250624 08:02:15.333525 28279 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 08:02:15.398666 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:15.398947 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:15.402098 28273 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 08:02:15.423192 28121 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "d73579fa2de84bda98141d07ff1c9b94"
tablet_id: "178abd11d618475d913d40de74d98c1a"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce"
}
peers {
permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94"
}
}
from {username='slave'} at 127.0.0.1:56928
W20250624 08:02:15.423585 28121 raft_consensus.cc:2216] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true
I20250624 08:02:15.423796 28121 raft_consensus.cc:3058] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:15.433830 28233 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" instance_seqno: 1750752134598573) as {username='slave'} at 127.26.229.196:57001; Asking this server to re-register.
I20250624 08:02:15.435103 28167 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:15.435295 28167 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:15.436419 28233 ts_manager.cc:194] Registered new tserver with Master: d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196:35095)
I20250624 08:02:16.130905 28233 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" instance_seqno: 1750752134091727) as {username='slave'} at 127.26.229.193:60777; Asking this server to re-register.
I20250624 08:02:16.131378 27774 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:16.131554 27774 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:16.132272 28233 ts_manager.cc:194] Registered new tserver with Master: a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193:36387)
I20250624 08:02:16.278659 27905 heartbeater.cc:344] Connected to a master server at 127.26.229.254:33439
I20250624 08:02:16.278837 27905 heartbeater.cc:499] Master 127.26.229.254:33439 was elected leader, sending a full tablet report...
I20250624 08:02:16.279399 28233 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eb5ee161f5504194823dd408d02f865a" instance_seqno: 1750752134257764) as {username='slave'} at 127.26.229.194:51767; Asking this server to re-register.
I20250624 08:02:16.279793 27905 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:16.279948 27905 heartbeater.cc:507] Master 127.26.229.254:33439 requested a full tablet report, sending...
I20250624 08:02:16.280462 28233 ts_manager.cc:194] Registered new tserver with Master: eb5ee161f5504194823dd408d02f865a (127.26.229.194:33395)
I20250624 08:02:16.620172 28315 raft_consensus.cc:491] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8f3360ae7f0436686505c1b7616c330)
I20250624 08:02:16.620291 28315 raft_consensus.cc:513] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } }
I20250624 08:02:16.621290 28315 leader_election.cc:290] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196:35095), a8f3360ae7f0436686505c1b7616c330 (127.26.229.195:44345)
W20250624 08:02:16.621992 27662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.229.195:44345: connect: Connection refused (error 111)
W20250624 08:02:16.622838 27662 leader_election.cc:336] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8f3360ae7f0436686505c1b7616c330 (127.26.229.195:44345): Network error: Client connection negotiation failed: client connection to 127.26.229.195:44345: connect: Connection refused (error 111)
I20250624 08:02:16.624466 28121 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "a759884bd8ab40a281916d17c4e9b6ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73579fa2de84bda98141d07ff1c9b94" is_pre_election: true
I20250624 08:02:16.624922 27664 leader_election.cc:304] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a759884bd8ab40a281916d17c4e9b6ce; no voters: a8f3360ae7f0436686505c1b7616c330, d73579fa2de84bda98141d07ff1c9b94
I20250624 08:02:16.625141 28315 raft_consensus.cc:2747] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250624 08:02:16.925386 28319 raft_consensus.cc:491] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250624 08:02:16.925559 28319 raft_consensus.cc:513] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true
I20250624 08:02:16.926429 28319 leader_election.cc:290] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193:36387)
I20250624 08:02:16.930038 27728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "d73579fa2de84bda98141d07ff1c9b94" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a759884bd8ab40a281916d17c4e9b6ce" is_pre_election: true
I20250624 08:02:16.930178 27728 raft_consensus.cc:2466] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73579fa2de84bda98141d07ff1c9b94 in term 1.
I20250624 08:02:16.930565 28056 leader_election.cc:304] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a759884bd8ab40a281916d17c4e9b6ce, d73579fa2de84bda98141d07ff1c9b94; no voters:
I20250624 08:02:16.930819 28319 raft_consensus.cc:2802] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250624 08:02:16.930877 28319 raft_consensus.cc:491] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250624 08:02:16.930908 28319 raft_consensus.cc:3058] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 2 FOLLOWER]: Advancing to term 3
I20250624 08:02:16.931764 28319 raft_consensus.cc:513] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true
I20250624 08:02:16.932044 28319 leader_election.cc:290] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [CANDIDATE]: Term 3 election: Requested vote from peers a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193:36387)
I20250624 08:02:16.932369 27728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178abd11d618475d913d40de74d98c1a" candidate_uuid: "d73579fa2de84bda98141d07ff1c9b94" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a759884bd8ab40a281916d17c4e9b6ce"
I20250624 08:02:16.932468 27728 raft_consensus.cc:3058] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 1 FOLLOWER]: Advancing to term 3
I20250624 08:02:16.933168 27728 raft_consensus.cc:2466] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73579fa2de84bda98141d07ff1c9b94 in term 3.
I20250624 08:02:16.933434 28056 leader_election.cc:304] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a759884bd8ab40a281916d17c4e9b6ce, d73579fa2de84bda98141d07ff1c9b94; no voters:
I20250624 08:02:16.933599 28319 raft_consensus.cc:2802] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 08:02:16.933760 28319 raft_consensus.cc:695] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 3 LEADER]: Becoming Leader. State: Replica: d73579fa2de84bda98141d07ff1c9b94, State: Running, Role: LEADER
I20250624 08:02:16.934065 28319 consensus_queue.cc:237] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true
I20250624 08:02:16.936043 28233 catalog_manager.cc:5582] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 reported cstate change: term changed from 1 to 3, leader changed from a8f3360ae7f0436686505c1b7616c330 (127.26.229.195) to d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196), now has a pending config: VOTER a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193), VOTER d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196). New cstate: current_term: 3 leader_uuid: "d73579fa2de84bda98141d07ff1c9b94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8f3360ae7f0436686505c1b7616c330" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 44345 } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true }
I20250624 08:02:17.409956 27728 raft_consensus.cc:1273] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Refusing update from remote peer d73579fa2de84bda98141d07ff1c9b94: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250624 08:02:17.410576 28319 consensus_queue.cc:1035] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250624 08:02:17.413173 28322 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER a8f3360ae7f0436686505c1b7616c330 (127.26.229.195) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true }
I20250624 08:02:17.413745 27728 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER a8f3360ae7f0436686505c1b7616c330 (127.26.229.195) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } unsafe_config_change: true }
I20250624 08:02:17.415805 28233 catalog_manager.cc:5582] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 reported cstate change: config changed from index -1 to 2, VOTER a8f3360ae7f0436686505c1b7616c330 (127.26.229.195) evicted, no longer has a pending config: VOTER a759884bd8ab40a281916d17c4e9b6ce (127.26.229.193), VOTER d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196). New cstate: current_term: 3 leader_uuid: "d73579fa2de84bda98141d07ff1c9b94" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250624 08:02:17.418053 28233 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 178abd11d618475d913d40de74d98c1a on TS a8f3360ae7f0436686505c1b7616c330: Not found: failed to reset TS proxy: Could not find TS for UUID a8f3360ae7f0436686505c1b7616c330
I20250624 08:02:17.424160 28121 consensus_queue.cc:237] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: true } } unsafe_config_change: true
I20250624 08:02:17.425421 27728 raft_consensus.cc:1273] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Refusing update from remote peer d73579fa2de84bda98141d07ff1c9b94: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250624 08:02:17.425833 28322 consensus_queue.cc:1035] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250624 08:02:17.427476 28319 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER eb5ee161f5504194823dd408d02f865a (127.26.229.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: true } } unsafe_config_change: true }
I20250624 08:02:17.427923 27728 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER eb5ee161f5504194823dd408d02f865a (127.26.229.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: true } } unsafe_config_change: true }
I20250624 08:02:17.429414 28222 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 178abd11d618475d913d40de74d98c1a with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250624 08:02:17.429876 28233 catalog_manager.cc:5582] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 reported cstate change: config changed from index 2 to 4, NON_VOTER eb5ee161f5504194823dd408d02f865a (127.26.229.194) added. New cstate: current_term: 3 leader_uuid: "d73579fa2de84bda98141d07ff1c9b94" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250624 08:02:17.430138 28055 consensus_peers.cc:487] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 -> Peer eb5ee161f5504194823dd408d02f865a (127.26.229.194:33395): Couldn't send request to peer eb5ee161f5504194823dd408d02f865a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 178abd11d618475d913d40de74d98c1a. This is attempt 1: this message will repeat every 5th retry.
W20250624 08:02:17.471252 28220 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 178abd11d618475d913d40de74d98c1a on TS a8f3360ae7f0436686505c1b7616c330 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a8f3360ae7f0436686505c1b7616c330
I20250624 08:02:17.811385 28336 ts_tablet_manager.cc:927] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Initiating tablet copy from peer d73579fa2de84bda98141d07ff1c9b94 (127.26.229.196:35095)
I20250624 08:02:17.811931 28336 tablet_copy_client.cc:323] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.229.196:35095
I20250624 08:02:17.815587 28141 tablet_copy_service.cc:140] P d73579fa2de84bda98141d07ff1c9b94: Received BeginTabletCopySession request for tablet 178abd11d618475d913d40de74d98c1a from peer eb5ee161f5504194823dd408d02f865a ({username='slave'} at 127.26.229.194:54241)
I20250624 08:02:17.815714 28141 tablet_copy_service.cc:161] P d73579fa2de84bda98141d07ff1c9b94: Beginning new tablet copy session on tablet 178abd11d618475d913d40de74d98c1a from peer eb5ee161f5504194823dd408d02f865a at {username='slave'} at 127.26.229.194:54241: session id = eb5ee161f5504194823dd408d02f865a-178abd11d618475d913d40de74d98c1a
I20250624 08:02:17.816828 28141 tablet_copy_source_session.cc:215] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 08:02:17.819700 28336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178abd11d618475d913d40de74d98c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:17.824126 28336 tablet_copy_client.cc:806] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: tablet copy: Starting download of 0 data blocks...
I20250624 08:02:17.824314 28336 tablet_copy_client.cc:670] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: tablet copy: Starting download of 1 WAL segments...
I20250624 08:02:17.825891 28336 tablet_copy_client.cc:538] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 08:02:17.827317 28336 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Bootstrap starting.
I20250624 08:02:17.831395 28336 log.cc:826] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:17.833787 28336 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:17.834355 28336 tablet_bootstrap.cc:492] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Bootstrap complete.
I20250624 08:02:17.834594 28336 ts_tablet_manager.cc:1397] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250624 08:02:17.836724 28336 raft_consensus.cc:357] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: true } } unsafe_config_change: true
I20250624 08:02:17.836902 28336 raft_consensus.cc:738] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: eb5ee161f5504194823dd408d02f865a, State: Initialized, Role: LEARNER
I20250624 08:02:17.837172 28336 consensus_queue.cc:260] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: true } } unsafe_config_change: true
I20250624 08:02:17.837989 28336 ts_tablet_manager.cc:1428] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:17.838692 28141 tablet_copy_service.cc:342] P d73579fa2de84bda98141d07ff1c9b94: Request end of tablet copy session eb5ee161f5504194823dd408d02f865a-178abd11d618475d913d40de74d98c1a received from {username='slave'} at 127.26.229.194:54241
I20250624 08:02:17.838840 28141 tablet_copy_service.cc:434] P d73579fa2de84bda98141d07ff1c9b94: ending tablet copy session eb5ee161f5504194823dd408d02f865a-178abd11d618475d913d40de74d98c1a on tablet 178abd11d618475d913d40de74d98c1a with peer eb5ee161f5504194823dd408d02f865a
I20250624 08:02:18.373510 27859 raft_consensus.cc:1215] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
W20250624 08:02:18.637180 28220 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 178abd11d618475d913d40de74d98c1a on TS a8f3360ae7f0436686505c1b7616c330 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a8f3360ae7f0436686505c1b7616c330
I20250624 08:02:18.759797 28341 raft_consensus.cc:1062] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94: attempting to promote NON_VOTER eb5ee161f5504194823dd408d02f865a to VOTER
I20250624 08:02:18.760242 28341 consensus_queue.cc:237] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: false } } unsafe_config_change: true
I20250624 08:02:18.761644 27728 raft_consensus.cc:1273] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Refusing update from remote peer d73579fa2de84bda98141d07ff1c9b94: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250624 08:02:18.761667 27859 raft_consensus.cc:1273] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a [term 3 LEARNER]: Refusing update from remote peer d73579fa2de84bda98141d07ff1c9b94: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250624 08:02:18.762030 28341 consensus_queue.cc:1035] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250624 08:02:18.762177 28322 consensus_queue.cc:1035] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250624 08:02:18.763954 28322 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, eb5ee161f5504194823dd408d02f865a (127.26.229.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: false } } unsafe_config_change: true }
I20250624 08:02:18.764366 27728 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P a759884bd8ab40a281916d17c4e9b6ce [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, eb5ee161f5504194823dd408d02f865a (127.26.229.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: false } } unsafe_config_change: true }
I20250624 08:02:18.765383 27859 raft_consensus.cc:2953] T 178abd11d618475d913d40de74d98c1a P eb5ee161f5504194823dd408d02f865a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, eb5ee161f5504194823dd408d02f865a (127.26.229.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: false } } unsafe_config_change: true }
I20250624 08:02:18.766396 28235 catalog_manager.cc:5582] T 178abd11d618475d913d40de74d98c1a P d73579fa2de84bda98141d07ff1c9b94 reported cstate change: config changed from index 4 to 5, eb5ee161f5504194823dd408d02f865a (127.26.229.194) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d73579fa2de84bda98141d07ff1c9b94" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 36387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94" member_type: VOTER last_known_addr { host: "127.26.229.196" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5ee161f5504194823dd408d02f865a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250624 08:02:18.807643 27543 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250624 08:02:18.817461 27543 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "178abd11d618475d913d40de74d98c1a"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "a759884bd8ab40a281916d17c4e9b6ce"
rpc_addresses {
host: "127.26.229.193"
port: 36387
}
}
ts_infos {
permanent_uuid: "d73579fa2de84bda98141d07ff1c9b94"
rpc_addresses {
host: "127.26.229.196"
port: 35095
}
}
ts_infos {
permanent_uuid: "eb5ee161f5504194823dd408d02f865a"
rpc_addresses {
host: "127.26.229.194"
port: 33395
}
}
I20250624 08:02:18.818645 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 27646
I20250624 08:02:18.824757 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 27777
I20250624 08:02:18.830163 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28039
I20250624 08:02:18.835529 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28203
2025-06-24T08:02:18Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (5212 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250624 08:02:18.858229 27543 test_util.cc:276] Using random seed: -1085042960
I20250624 08:02:18.859221 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:18.859270 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:18.859288 27543 ts_itest-base.cc:117] 3 tablet servers
I20250624 08:02:18.859304 27543 ts_itest-base.cc:118] 3 replicas per TS
I20250624 08:02:18.859331 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:18Z Disabled control of system clock
I20250624 08:02:18.867120 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:38883
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:38883
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250624 08:02:18.964762 28361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:18.964994 28361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:18.965062 28361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:18.968129 28361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:18.968186 28361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:18.968219 28361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:18.968264 28361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:18.971906 28361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:38883
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.28361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:18.972963 28361 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:18.974079 28361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:18.981101 28369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:18.981101 28366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:18.981511 28361 server_base.cc:1048] running on GCE node
W20250624 08:02:18.981534 28367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:18.982110 28361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:18.983161 28361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:18.987141 28361 hybrid_clock.cc:648] HybridClock initialized: now 1750752138987117 us; error 38 us; skew 500 ppm
I20250624 08:02:18.989351 28361 webserver.cc:469] Webserver started at http://127.26.229.254:33759/ using document root <none> and password file <none>
I20250624 08:02:18.989959 28361 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:18.990021 28361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:18.990212 28361 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:18.992102 28361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc"
format_stamp: "Formatted at 2025-06-24 08:02:18 on dist-test-slave-4j8z"
I20250624 08:02:18.992626 28361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc"
format_stamp: "Formatted at 2025-06-24 08:02:18 on dist-test-slave-4j8z"
I20250624 08:02:18.996057 28361 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 08:02:18.998208 28375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:18.999236 28361 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:18.999363 28361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc"
format_stamp: "Formatted at 2025-06-24 08:02:18 on dist-test-slave-4j8z"
I20250624 08:02:18.999449 28361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:19.016853 28361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:19.017525 28361 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:19.017704 28361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:19.024638 28361 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:38883
I20250624 08:02:19.024644 28427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:38883 every 8 connection(s)
I20250624 08:02:19.025584 28361 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:19.028028 28428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:19.032123 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28361
I20250624 08:02:19.032229 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:19.032826 28428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc: Bootstrap starting.
I20250624 08:02:19.034961 28428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:19.035794 28428 log.cc:826] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:19.037451 28428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc: No bootstrap required, opened a new log
I20250624 08:02:19.040001 28428 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 38883 } }
I20250624 08:02:19.040202 28428 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:19.040266 28428 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8321bf5da5ff4ed6bde2954a1c2c7fdc, State: Initialized, Role: FOLLOWER
I20250624 08:02:19.040685 28428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 38883 } }
I20250624 08:02:19.040829 28428 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:19.040897 28428 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:19.040987 28428 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:19.041677 28428 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 38883 } }
I20250624 08:02:19.041975 28428 leader_election.cc:304] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8321bf5da5ff4ed6bde2954a1c2c7fdc; no voters:
I20250624 08:02:19.042209 28428 leader_election.cc:290] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:19.042316 28433 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:19.042518 28433 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [term 1 LEADER]: Becoming Leader. State: Replica: 8321bf5da5ff4ed6bde2954a1c2c7fdc, State: Running, Role: LEADER
I20250624 08:02:19.042832 28433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 38883 } }
I20250624 08:02:19.043236 28428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:19.044384 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 38883 } } }
I20250624 08:02:19.044445 28435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8321bf5da5ff4ed6bde2954a1c2c7fdc. Latest consensus state: current_term: 1 leader_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8321bf5da5ff4ed6bde2954a1c2c7fdc" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 38883 } } }
I20250624 08:02:19.044632 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:19.044646 28435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:19.045372 28444 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:19.048342 28444 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:19.052706 28444 catalog_manager.cc:1349] Generated new cluster ID: e08019dc56484292bf8cd724a55375bf
I20250624 08:02:19.052767 28444 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:19.058383 28444 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:19.059572 28444 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:19.066843 28444 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8321bf5da5ff4ed6bde2954a1c2c7fdc: Generated new TSK 0
I20250624 08:02:19.067591 28444 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:19.074311 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:38883
--builtin_ntp_servers=127.26.229.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250624 08:02:19.177867 28452 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 08:02:19.178095 28452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:19.178135 28452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:19.178206 28452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:19.181425 28452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:19.181557 28452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:19.185108 28452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:38883
--never_fsync=true
--heap_profile_path=/tmp/kudu.28452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:19.186085 28452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:19.187186 28452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:19.194972 28457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:19.195035 28452 server_base.cc:1048] running on GCE node
W20250624 08:02:19.195142 28458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:19.195308 28460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:19.195688 28452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:19.196318 28452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:19.197499 28452 hybrid_clock.cc:648] HybridClock initialized: now 1750752139197480 us; error 34 us; skew 500 ppm
I20250624 08:02:19.199692 28452 webserver.cc:469] Webserver started at http://127.26.229.193:36645/ using document root <none> and password file <none>
I20250624 08:02:19.200388 28452 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:19.200480 28452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:19.200693 28452 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:19.202821 28452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "07c9b2e1cd0a4e82999640a3af7a42d0"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.203600 28452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "07c9b2e1cd0a4e82999640a3af7a42d0"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.207945 28452 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250624 08:02:19.210418 28466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.211534 28452 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250624 08:02:19.211678 28452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "07c9b2e1cd0a4e82999640a3af7a42d0"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.211783 28452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:19.225242 28452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:19.225782 28452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:19.225926 28452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:19.226365 28452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:19.227254 28452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:19.227301 28452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.227341 28452 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:19.227365 28452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.236119 28452 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:43399
I20250624 08:02:19.236145 28579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:43399 every 8 connection(s)
I20250624 08:02:19.236979 28452 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:19.240056 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28452
I20250624 08:02:19.240139 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:19.241995 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:38883
--builtin_ntp_servers=127.26.229.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 08:02:19.245663 28580 heartbeater.cc:344] Connected to a master server at 127.26.229.254:38883
I20250624 08:02:19.245908 28580 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:19.246539 28580 heartbeater.cc:507] Master 127.26.229.254:38883 requested a full tablet report, sending...
I20250624 08:02:19.247908 28392 ts_manager.cc:194] Registered new tserver with Master: 07c9b2e1cd0a4e82999640a3af7a42d0 (127.26.229.193:43399)
I20250624 08:02:19.248986 28392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:43289
W20250624 08:02:19.338755 28583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 08:02:19.338965 28583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:19.338996 28583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:19.339059 28583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:19.341995 28583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:19.342125 28583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:19.346254 28583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:38883
--never_fsync=true
--heap_profile_path=/tmp/kudu.28583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:19.347201 28583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:19.348232 28583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:19.355340 28591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:19.355280 28589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:19.355310 28588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:19.355557 28583 server_base.cc:1048] running on GCE node
I20250624 08:02:19.355868 28583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:19.356313 28583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:19.357450 28583 hybrid_clock.cc:648] HybridClock initialized: now 1750752139357440 us; error 23 us; skew 500 ppm
I20250624 08:02:19.359112 28583 webserver.cc:469] Webserver started at http://127.26.229.194:42483/ using document root <none> and password file <none>
I20250624 08:02:19.359622 28583 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:19.359673 28583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:19.359859 28583 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:19.361316 28583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f9b0562e720b45eaaa93830f4c185eee"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.361752 28583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f9b0562e720b45eaaa93830f4c185eee"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.364878 28583 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:19.366755 28597 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.367719 28583 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:19.367802 28583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f9b0562e720b45eaaa93830f4c185eee"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.367866 28583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:19.384238 28583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:19.384737 28583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:19.384856 28583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:19.385279 28583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:19.386036 28583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:19.386078 28583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.386112 28583 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:19.386132 28583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.394695 28583 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:38215
I20250624 08:02:19.394722 28710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:38215 every 8 connection(s)
I20250624 08:02:19.395568 28583 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:19.397495 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28583
I20250624 08:02:19.397576 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 08:02:19.399703 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:38883
--builtin_ntp_servers=127.26.229.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 08:02:19.404306 28711 heartbeater.cc:344] Connected to a master server at 127.26.229.254:38883
I20250624 08:02:19.404513 28711 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:19.404963 28711 heartbeater.cc:507] Master 127.26.229.254:38883 requested a full tablet report, sending...
I20250624 08:02:19.405784 28392 ts_manager.cc:194] Registered new tserver with Master: f9b0562e720b45eaaa93830f4c185eee (127.26.229.194:38215)
I20250624 08:02:19.406265 28392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:42885
W20250624 08:02:19.497917 28714 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 08:02:19.498129 28714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:19.498193 28714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:19.498265 28714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:19.501463 28714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:19.501598 28714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:19.505216 28714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:38883
--never_fsync=true
--heap_profile_path=/tmp/kudu.28714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:19.506208 28714 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:19.507297 28714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:19.514384 28722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:19.514627 28719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:19.514698 28720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:19.514899 28714 server_base.cc:1048] running on GCE node
I20250624 08:02:19.515252 28714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:19.515766 28714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:19.516938 28714 hybrid_clock.cc:648] HybridClock initialized: now 1750752139516922 us; error 24 us; skew 500 ppm
I20250624 08:02:19.518797 28714 webserver.cc:469] Webserver started at http://127.26.229.195:32867/ using document root <none> and password file <none>
I20250624 08:02:19.519366 28714 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:19.519428 28714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:19.519611 28714 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:19.521133 28714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "112158cf96e243c194ad0b0b1e20e200"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.521588 28714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "112158cf96e243c194ad0b0b1e20e200"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.524684 28714 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:19.526695 28728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.527643 28714 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:19.527752 28714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "112158cf96e243c194ad0b0b1e20e200"
format_stamp: "Formatted at 2025-06-24 08:02:19 on dist-test-slave-4j8z"
I20250624 08:02:19.527830 28714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:19.540889 28714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:19.541453 28714 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:19.541608 28714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:19.542085 28714 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:19.542997 28714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:19.543071 28714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.543126 28714 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:19.543156 28714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:19.551518 28714 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:34889
I20250624 08:02:19.551545 28841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:34889 every 8 connection(s)
I20250624 08:02:19.552384 28714 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:19.555279 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28714
I20250624 08:02:19.555392 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 08:02:19.561020 28842 heartbeater.cc:344] Connected to a master server at 127.26.229.254:38883
I20250624 08:02:19.561249 28842 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:19.561733 28842 heartbeater.cc:507] Master 127.26.229.254:38883 requested a full tablet report, sending...
I20250624 08:02:19.562580 28392 ts_manager.cc:194] Registered new tserver with Master: 112158cf96e243c194ad0b0b1e20e200 (127.26.229.195:34889)
I20250624 08:02:19.563127 28392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:37401
I20250624 08:02:19.568976 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:19.576928 28392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58244:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 08:02:19.579283 28392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:02:19.593967 28645 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf43e85f76f41ed82dd85feb98488ed (DEFAULT_TABLE table=TestTable [id=38fa194f1cec4f19b4d1e20a330e7411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:19.593968 28514 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf43e85f76f41ed82dd85feb98488ed (DEFAULT_TABLE table=TestTable [id=38fa194f1cec4f19b4d1e20a330e7411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:19.593968 28776 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf43e85f76f41ed82dd85feb98488ed (DEFAULT_TABLE table=TestTable [id=38fa194f1cec4f19b4d1e20a330e7411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:19.594851 28645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf43e85f76f41ed82dd85feb98488ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:19.594854 28776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf43e85f76f41ed82dd85feb98488ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:19.594854 28514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf43e85f76f41ed82dd85feb98488ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:19.599573 28859 tablet_bootstrap.cc:492] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: Bootstrap starting.
I20250624 08:02:19.600786 28861 tablet_bootstrap.cc:492] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: Bootstrap starting.
I20250624 08:02:19.600869 28860 tablet_bootstrap.cc:492] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: Bootstrap starting.
I20250624 08:02:19.601979 28859 tablet_bootstrap.cc:654] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:19.602564 28861 tablet_bootstrap.cc:654] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:19.602821 28860 tablet_bootstrap.cc:654] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:19.602823 28859 log.cc:826] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:19.603238 28861 log.cc:826] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:19.603520 28860 log.cc:826] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:19.604604 28859 tablet_bootstrap.cc:492] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: No bootstrap required, opened a new log
I20250624 08:02:19.604665 28861 tablet_bootstrap.cc:492] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: No bootstrap required, opened a new log
I20250624 08:02:19.604786 28859 ts_tablet_manager.cc:1397] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250624 08:02:19.604810 28861 ts_tablet_manager.cc:1397] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 08:02:19.605038 28860 tablet_bootstrap.cc:492] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: No bootstrap required, opened a new log
I20250624 08:02:19.605212 28860 ts_tablet_manager.cc:1397] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 08:02:19.607055 28861 raft_consensus.cc:357] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.607245 28861 raft_consensus.cc:738] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c9b2e1cd0a4e82999640a3af7a42d0, State: Initialized, Role: FOLLOWER
I20250624 08:02:19.607566 28861 consensus_queue.cc:260] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.607700 28859 raft_consensus.cc:357] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.607929 28859 raft_consensus.cc:738] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b0562e720b45eaaa93830f4c185eee, State: Initialized, Role: FOLLOWER
I20250624 08:02:19.607853 28860 raft_consensus.cc:357] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.608022 28860 raft_consensus.cc:738] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 112158cf96e243c194ad0b0b1e20e200, State: Initialized, Role: FOLLOWER
I20250624 08:02:19.608232 28859 consensus_queue.cc:260] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.608273 28860 consensus_queue.cc:260] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.608395 28861 ts_tablet_manager.cc:1428] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 08:02:19.608742 28580 heartbeater.cc:499] Master 127.26.229.254:38883 was elected leader, sending a full tablet report...
I20250624 08:02:19.609007 28859 ts_tablet_manager.cc:1428] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:19.609007 28860 ts_tablet_manager.cc:1428] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 08:02:19.609364 28711 heartbeater.cc:499] Master 127.26.229.254:38883 was elected leader, sending a full tablet report...
I20250624 08:02:19.610189 28842 heartbeater.cc:499] Master 127.26.229.254:38883 was elected leader, sending a full tablet report...
I20250624 08:02:19.617889 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:19.619158 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 07c9b2e1cd0a4e82999640a3af7a42d0 to finish bootstrapping
I20250624 08:02:19.624402 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f9b0562e720b45eaaa93830f4c185eee to finish bootstrapping
I20250624 08:02:19.628993 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 112158cf96e243c194ad0b0b1e20e200 to finish bootstrapping
I20250624 08:02:19.643945 28665 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed"
dest_uuid: "f9b0562e720b45eaaa93830f4c185eee"
from {username='slave'} at 127.0.0.1:45966
I20250624 08:02:19.644142 28665 raft_consensus.cc:491] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 08:02:19.644274 28665 raft_consensus.cc:3058] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:19.645213 28665 raft_consensus.cc:513] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.646163 28665 leader_election.cc:290] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [CANDIDATE]: Term 1 election: Requested vote from peers 07c9b2e1cd0a4e82999640a3af7a42d0 (127.26.229.193:43399), 112158cf96e243c194ad0b0b1e20e200 (127.26.229.195:34889)
W20250624 08:02:19.646323 28712 tablet.cc:2378] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:19.649071 27543 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250624 08:02:19.650058 28796 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed" candidate_uuid: "f9b0562e720b45eaaa93830f4c185eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "112158cf96e243c194ad0b0b1e20e200"
I20250624 08:02:19.650259 28796 raft_consensus.cc:3058] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:19.651086 28534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed" candidate_uuid: "f9b0562e720b45eaaa93830f4c185eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0"
I20250624 08:02:19.651274 28534 raft_consensus.cc:3058] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:19.651367 28796 raft_consensus.cc:2466] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b0562e720b45eaaa93830f4c185eee in term 1.
I20250624 08:02:19.651762 28600 leader_election.cc:304] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 112158cf96e243c194ad0b0b1e20e200, f9b0562e720b45eaaa93830f4c185eee; no voters:
I20250624 08:02:19.652024 28867 raft_consensus.cc:2802] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:19.652230 28867 raft_consensus.cc:695] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 LEADER]: Becoming Leader. State: Replica: f9b0562e720b45eaaa93830f4c185eee, State: Running, Role: LEADER
I20250624 08:02:19.652335 28534 raft_consensus.cc:2466] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b0562e720b45eaaa93830f4c185eee in term 1.
I20250624 08:02:19.652572 28867 consensus_queue.cc:237] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:19.655354 28390 catalog_manager.cc:5582] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee reported cstate change: term changed from 0 to 1, leader changed from <none> to f9b0562e720b45eaaa93830f4c185eee (127.26.229.194). New cstate: current_term: 1 leader_uuid: "f9b0562e720b45eaaa93830f4c185eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } health_report { overall_health: HEALTHY } } }
W20250624 08:02:19.738199 28581 tablet.cc:2378] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:19.751825 27543 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250624 08:02:19.803251 28843 tablet.cc:2378] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:19.954452 27543 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250624 08:02:20.119297 28867 consensus_queue.cc:1035] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:20.124365 28876 consensus_queue.cc:1035] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:20.389443 28665 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed"
dest_uuid: "f9b0562e720b45eaaa93830f4c185eee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45978
I20250624 08:02:20.389597 28665 raft_consensus.cc:604] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 LEADER]: Received request to transfer leadership
I20250624 08:02:20.551472 28867 raft_consensus.cc:991] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee: : Instructing follower 112158cf96e243c194ad0b0b1e20e200 to start an election
I20250624 08:02:20.551604 28867 raft_consensus.cc:1079] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 LEADER]: Signalling peer 112158cf96e243c194ad0b0b1e20e200 to start an election
I20250624 08:02:20.552155 28796 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed"
dest_uuid: "112158cf96e243c194ad0b0b1e20e200"
from {username='slave'} at 127.26.229.194:59593
I20250624 08:02:20.552322 28796 raft_consensus.cc:491] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 08:02:20.552398 28796 raft_consensus.cc:3058] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:20.553169 28796 raft_consensus.cc:513] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:20.554070 28796 leader_election.cc:290] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [CANDIDATE]: Term 2 election: Requested vote from peers 07c9b2e1cd0a4e82999640a3af7a42d0 (127.26.229.193:43399), f9b0562e720b45eaaa93830f4c185eee (127.26.229.194:38215)
I20250624 08:02:20.558411 28534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed" candidate_uuid: "112158cf96e243c194ad0b0b1e20e200" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0"
I20250624 08:02:20.558557 28534 raft_consensus.cc:3058] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:20.558712 28665 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed" candidate_uuid: "112158cf96e243c194ad0b0b1e20e200" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9b0562e720b45eaaa93830f4c185eee"
I20250624 08:02:20.558883 28665 raft_consensus.cc:3053] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 LEADER]: Stepping down as leader of term 1
I20250624 08:02:20.558918 28665 raft_consensus.cc:738] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9b0562e720b45eaaa93830f4c185eee, State: Running, Role: LEADER
I20250624 08:02:20.559042 28665 consensus_queue.cc:260] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:20.559257 28665 raft_consensus.cc:3058] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:20.559532 28534 raft_consensus.cc:2466] T 2cf43e85f76f41ed82dd85feb98488ed P 07c9b2e1cd0a4e82999640a3af7a42d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 112158cf96e243c194ad0b0b1e20e200 in term 2.
I20250624 08:02:20.559935 28729 leader_election.cc:304] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c9b2e1cd0a4e82999640a3af7a42d0, 112158cf96e243c194ad0b0b1e20e200; no voters:
I20250624 08:02:20.560098 28665 raft_consensus.cc:2466] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 112158cf96e243c194ad0b0b1e20e200 in term 2.
I20250624 08:02:20.560165 28878 raft_consensus.cc:2802] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:20.560362 28878 raft_consensus.cc:695] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [term 2 LEADER]: Becoming Leader. State: Replica: 112158cf96e243c194ad0b0b1e20e200, State: Running, Role: LEADER
I20250624 08:02:20.560676 28878 consensus_queue.cc:237] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } }
I20250624 08:02:20.563001 28392 catalog_manager.cc:5582] T 2cf43e85f76f41ed82dd85feb98488ed P 112158cf96e243c194ad0b0b1e20e200 reported cstate change: term changed from 1 to 2, leader changed from f9b0562e720b45eaaa93830f4c185eee (127.26.229.194) to 112158cf96e243c194ad0b0b1e20e200 (127.26.229.195). New cstate: current_term: 2 leader_uuid: "112158cf96e243c194ad0b0b1e20e200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c9b2e1cd0a4e82999640a3af7a42d0" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "112158cf96e243c194ad0b0b1e20e200" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 34889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b0562e720b45eaaa93830f4c185eee" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38215 } health_report { overall_health: UNKNOWN } } }
I20250624 08:02:20.797547 28665 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2cf43e85f76f41ed82dd85feb98488ed"
dest_uuid: "f9b0562e720b45eaaa93830f4c185eee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45992
I20250624 08:02:20.797681 28665 raft_consensus.cc:604] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 2 FOLLOWER]: Received request to transfer leadership
I20250624 08:02:20.797744 28665 raft_consensus.cc:612] T 2cf43e85f76f41ed82dd85feb98488ed P f9b0562e720b45eaaa93830f4c185eee [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250624 08:02:20.806139 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28452
I20250624 08:02:20.811654 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28583
I20250624 08:02:20.817236 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28714
I20250624 08:02:20.822669 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28361
2025-06-24T08:02:20Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (1985 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250624 08:02:20.844062 27543 test_util.cc:276] Using random seed: -1083057127
I20250624 08:02:20.844797 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:20.844846 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:20.844858 27543 ts_itest-base.cc:117] 3 tablet servers
I20250624 08:02:20.844869 27543 ts_itest-base.cc:118] 3 replicas per TS
I20250624 08:02:20.844884 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:20Z Disabled control of system clock
I20250624 08:02:20.852658 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:43807
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:43807 with env {}
W20250624 08:02:20.951184 28927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:20.951413 28927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:20.951483 28927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:20.954535 28927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:20.954588 28927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:20.954620 28927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:20.954650 28927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:20.958163 28927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:43807
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.28927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:20.959127 28927 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:20.960196 28927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250624 08:02:20.966837 28927 server_base.cc:1048] running on GCE node
W20250624 08:02:20.966769 28935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:20.966859 28933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:20.966864 28932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:20.967519 28927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:20.968358 28927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:20.969509 28927 hybrid_clock.cc:648] HybridClock initialized: now 1750752140969494 us; error 51 us; skew 500 ppm
I20250624 08:02:20.971277 28927 webserver.cc:469] Webserver started at http://127.26.229.254:33787/ using document root <none> and password file <none>
I20250624 08:02:20.971812 28927 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:20.971876 28927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:20.972050 28927 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:20.973572 28927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "df2dcd002b4140bea0e300aa24bb21c4"
format_stamp: "Formatted at 2025-06-24 08:02:20 on dist-test-slave-4j8z"
I20250624 08:02:20.974031 28927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "df2dcd002b4140bea0e300aa24bb21c4"
format_stamp: "Formatted at 2025-06-24 08:02:20 on dist-test-slave-4j8z"
I20250624 08:02:20.977409 28927 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250624 08:02:20.979485 28941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:20.980408 28927 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:20.980525 28927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "df2dcd002b4140bea0e300aa24bb21c4"
format_stamp: "Formatted at 2025-06-24 08:02:20 on dist-test-slave-4j8z"
I20250624 08:02:20.980607 28927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:20.998934 28927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:20.999511 28927 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:20.999668 28927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:21.006042 28927 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:43807
I20250624 08:02:21.006058 28993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:43807 every 8 connection(s)
I20250624 08:02:21.006968 28927 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:21.007123 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 28927
I20250624 08:02:21.007241 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:21.009912 28994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.015436 28994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4: Bootstrap starting.
I20250624 08:02:21.017494 28994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.018136 28994 log.cc:826] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:21.019731 28994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4: No bootstrap required, opened a new log
I20250624 08:02:21.021926 28994 raft_consensus.cc:357] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2dcd002b4140bea0e300aa24bb21c4" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 43807 } }
I20250624 08:02:21.022087 28994 raft_consensus.cc:383] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.022135 28994 raft_consensus.cc:738] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2dcd002b4140bea0e300aa24bb21c4, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.022449 28994 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2dcd002b4140bea0e300aa24bb21c4" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 43807 } }
I20250624 08:02:21.022554 28994 raft_consensus.cc:397] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:21.022591 28994 raft_consensus.cc:491] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:21.022667 28994 raft_consensus.cc:3058] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:21.023317 28994 raft_consensus.cc:513] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2dcd002b4140bea0e300aa24bb21c4" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 43807 } }
I20250624 08:02:21.023568 28994 leader_election.cc:304] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2dcd002b4140bea0e300aa24bb21c4; no voters:
I20250624 08:02:21.023770 28994 leader_election.cc:290] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:21.023874 28999 raft_consensus.cc:2802] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:21.024052 28999 raft_consensus.cc:695] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [term 1 LEADER]: Becoming Leader. State: Replica: df2dcd002b4140bea0e300aa24bb21c4, State: Running, Role: LEADER
I20250624 08:02:21.024340 28999 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2dcd002b4140bea0e300aa24bb21c4" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 43807 } }
I20250624 08:02:21.024729 28994 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:21.025914 29000 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2dcd002b4140bea0e300aa24bb21c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2dcd002b4140bea0e300aa24bb21c4" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 43807 } } }
I20250624 08:02:21.026008 29000 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:21.026021 29001 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2dcd002b4140bea0e300aa24bb21c4. Latest consensus state: current_term: 1 leader_uuid: "df2dcd002b4140bea0e300aa24bb21c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2dcd002b4140bea0e300aa24bb21c4" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 43807 } } }
I20250624 08:02:21.026116 29001 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:21.026770 29008 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:21.028954 29008 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:21.033854 29008 catalog_manager.cc:1349] Generated new cluster ID: 28fbe1b6361149e68ed30a026039cf37
I20250624 08:02:21.033905 29008 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:21.046485 29008 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:21.047271 29008 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:21.055190 29008 catalog_manager.cc:5955] T 00000000000000000000000000000000 P df2dcd002b4140bea0e300aa24bb21c4: Generated new TSK 0
I20250624 08:02:21.055768 29008 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:21.063089 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:43807
--builtin_ntp_servers=127.26.229.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 08:02:21.166123 29018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:21.166296 29018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:21.166337 29018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:21.169445 29018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:21.169538 29018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:21.173003 29018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:43807
--never_fsync=true
--heap_profile_path=/tmp/kudu.29018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:21.173936 29018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:21.175014 29018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:21.182580 29024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:21.182565 29023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:21.182561 29026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:21.182893 29018 server_base.cc:1048] running on GCE node
I20250624 08:02:21.183384 29018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:21.183956 29018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:21.185127 29018 hybrid_clock.cc:648] HybridClock initialized: now 1750752141185113 us; error 26 us; skew 500 ppm
I20250624 08:02:21.187129 29018 webserver.cc:469] Webserver started at http://127.26.229.193:36943/ using document root <none> and password file <none>
I20250624 08:02:21.187728 29018 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:21.187794 29018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:21.187969 29018 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:21.189841 29018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f202001d5b444ce9926562c65505d5a5"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.190289 29018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f202001d5b444ce9926562c65505d5a5"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.193542 29018 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250624 08:02:21.195448 29032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.196348 29018 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:21.196445 29018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f202001d5b444ce9926562c65505d5a5"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.196521 29018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:21.210340 29018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:21.210943 29018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:21.211105 29018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:21.211598 29018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:21.212430 29018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:21.212478 29018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.212536 29018 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:21.212569 29018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.220790 29018 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:45471
I20250624 08:02:21.220830 29145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:45471 every 8 connection(s)
I20250624 08:02:21.221668 29018 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:21.228449 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29018
I20250624 08:02:21.228543 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:21.230329 29146 heartbeater.cc:344] Connected to a master server at 127.26.229.254:43807
I20250624 08:02:21.230501 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:43807
--builtin_ntp_servers=127.26.229.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:21.230545 29146 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:21.231096 29146 heartbeater.cc:507] Master 127.26.229.254:43807 requested a full tablet report, sending...
I20250624 08:02:21.232357 28957 ts_manager.cc:194] Registered new tserver with Master: f202001d5b444ce9926562c65505d5a5 (127.26.229.193:45471)
I20250624 08:02:21.233541 28957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:44275
W20250624 08:02:21.328996 29149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:21.329195 29149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:21.329234 29149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:21.332358 29149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:21.332468 29149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:21.336046 29149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:43807
--never_fsync=true
--heap_profile_path=/tmp/kudu.29149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:21.336980 29149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:21.338004 29149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:21.345085 29154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:21.345121 29155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:21.345120 29157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:21.345367 29149 server_base.cc:1048] running on GCE node
I20250624 08:02:21.345674 29149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:21.346134 29149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:21.347280 29149 hybrid_clock.cc:648] HybridClock initialized: now 1750752141347266 us; error 26 us; skew 500 ppm
I20250624 08:02:21.348948 29149 webserver.cc:469] Webserver started at http://127.26.229.194:41415/ using document root <none> and password file <none>
I20250624 08:02:21.349453 29149 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:21.349505 29149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:21.349673 29149 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:21.351173 29149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9a5d0cfc3462438995fa4abeca06cd15"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.351620 29149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9a5d0cfc3462438995fa4abeca06cd15"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.354514 29149 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250624 08:02:21.356436 29163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.357322 29149 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:21.357414 29149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9a5d0cfc3462438995fa4abeca06cd15"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.357506 29149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:21.367414 29149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:21.367882 29149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:21.368017 29149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:21.368458 29149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:21.369220 29149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:21.369263 29149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.369313 29149 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:21.369336 29149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.377436 29149 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:38169
I20250624 08:02:21.377457 29276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:38169 every 8 connection(s)
I20250624 08:02:21.378217 29149 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:21.385597 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29149
I20250624 08:02:21.385692 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 08:02:21.385838 29277 heartbeater.cc:344] Connected to a master server at 127.26.229.254:43807
I20250624 08:02:21.386076 29277 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:21.386629 29277 heartbeater.cc:507] Master 127.26.229.254:43807 requested a full tablet report, sending...
I20250624 08:02:21.387585 28957 ts_manager.cc:194] Registered new tserver with Master: 9a5d0cfc3462438995fa4abeca06cd15 (127.26.229.194:38169)
I20250624 08:02:21.387835 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:43807
--builtin_ntp_servers=127.26.229.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:21.388156 28957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:39559
W20250624 08:02:21.485862 29280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:21.486063 29280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:21.486140 29280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:21.489168 29280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:21.489300 29280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:21.492838 29280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:43807
--never_fsync=true
--heap_profile_path=/tmp/kudu.29280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:21.493880 29280 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:21.494966 29280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:21.502167 29285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:21.502182 29288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:21.502090 29286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:21.502439 29280 server_base.cc:1048] running on GCE node
I20250624 08:02:21.502763 29280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:21.503297 29280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:21.504442 29280 hybrid_clock.cc:648] HybridClock initialized: now 1750752141504414 us; error 40 us; skew 500 ppm
I20250624 08:02:21.506179 29280 webserver.cc:469] Webserver started at http://127.26.229.195:35571/ using document root <none> and password file <none>
I20250624 08:02:21.506702 29280 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:21.506757 29280 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:21.506965 29280 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:21.508498 29280 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.508934 29280 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.512009 29280 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250624 08:02:21.514019 29294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.515029 29280 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250624 08:02:21.515111 29280 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e"
format_stamp: "Formatted at 2025-06-24 08:02:21 on dist-test-slave-4j8z"
I20250624 08:02:21.515223 29280 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:21.539002 29280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:21.539633 29280 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:21.539808 29280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:21.540319 29280 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:21.541200 29280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:21.541249 29280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.541311 29280 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:21.541338 29280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:21.549657 29280 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:39975
I20250624 08:02:21.549685 29407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:39975 every 8 connection(s)
I20250624 08:02:21.550510 29280 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:21.553185 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29280
I20250624 08:02:21.553284 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 08:02:21.559177 29408 heartbeater.cc:344] Connected to a master server at 127.26.229.254:43807
I20250624 08:02:21.559367 29408 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:21.559800 29408 heartbeater.cc:507] Master 127.26.229.254:43807 requested a full tablet report, sending...
I20250624 08:02:21.560647 28957 ts_manager.cc:194] Registered new tserver with Master: 93c1cfb1ffbf4591bb2e193cc63d9e5e (127.26.229.195:39975)
I20250624 08:02:21.561206 28957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:50969
I20250624 08:02:21.567134 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:21.575285 28957 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43998:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 08:02:21.578193 28957 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:02:21.593457 29342 tablet_service.cc:1468] Processing CreateTablet for tablet 541528f8db4946c2bbd99c2f7c66a681 (DEFAULT_TABLE table=TestTable [id=819e178f46114aeaa83a71a98daf0ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:21.593492 29080 tablet_service.cc:1468] Processing CreateTablet for tablet 541528f8db4946c2bbd99c2f7c66a681 (DEFAULT_TABLE table=TestTable [id=819e178f46114aeaa83a71a98daf0ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:21.593492 29211 tablet_service.cc:1468] Processing CreateTablet for tablet 541528f8db4946c2bbd99c2f7c66a681 (DEFAULT_TABLE table=TestTable [id=819e178f46114aeaa83a71a98daf0ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:21.594487 29342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541528f8db4946c2bbd99c2f7c66a681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.594496 29080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541528f8db4946c2bbd99c2f7c66a681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.594497 29211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541528f8db4946c2bbd99c2f7c66a681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.599155 29425 tablet_bootstrap.cc:492] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Bootstrap starting.
I20250624 08:02:21.599741 29426 tablet_bootstrap.cc:492] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: Bootstrap starting.
I20250624 08:02:21.600386 29427 tablet_bootstrap.cc:492] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: Bootstrap starting.
I20250624 08:02:21.602020 29426 tablet_bootstrap.cc:654] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.602020 29425 tablet_bootstrap.cc:654] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.602169 29427 tablet_bootstrap.cc:654] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.602841 29427 log.cc:826] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:21.602885 29426 log.cc:826] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:21.602885 29425 log.cc:826] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:21.604283 29427 tablet_bootstrap.cc:492] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: No bootstrap required, opened a new log
I20250624 08:02:21.604398 29426 tablet_bootstrap.cc:492] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: No bootstrap required, opened a new log
I20250624 08:02:21.604398 29425 tablet_bootstrap.cc:492] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: No bootstrap required, opened a new log
I20250624 08:02:21.604470 29427 ts_tablet_manager.cc:1397] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 08:02:21.604566 29425 ts_tablet_manager.cc:1397] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250624 08:02:21.604566 29426 ts_tablet_manager.cc:1397] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250624 08:02:21.606542 29427 raft_consensus.cc:357] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.606695 29427 raft_consensus.cc:383] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.606768 29427 raft_consensus.cc:738] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f202001d5b444ce9926562c65505d5a5, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.607182 29427 consensus_queue.cc:260] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.607259 29426 raft_consensus.cc:357] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.607504 29426 raft_consensus.cc:383] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.607602 29426 raft_consensus.cc:738] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5d0cfc3462438995fa4abeca06cd15, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.607738 29146 heartbeater.cc:499] Master 127.26.229.254:43807 was elected leader, sending a full tablet report...
I20250624 08:02:21.607972 29427 ts_tablet_manager.cc:1428] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250624 08:02:21.607898 29425 raft_consensus.cc:357] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.608100 29425 raft_consensus.cc:383] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.608155 29425 raft_consensus.cc:738] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c1cfb1ffbf4591bb2e193cc63d9e5e, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.608080 29426 consensus_queue.cc:260] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.608513 29425 consensus_queue.cc:260] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.608819 29426 ts_tablet_manager.cc:1428] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:21.609179 29277 heartbeater.cc:499] Master 127.26.229.254:43807 was elected leader, sending a full tablet report...
I20250624 08:02:21.609584 29408 heartbeater.cc:499] Master 127.26.229.254:43807 was elected leader, sending a full tablet report...
I20250624 08:02:21.611029 29425 ts_tablet_manager.cc:1428] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20250624 08:02:21.629036 29278 tablet.cc:2378] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:21.642429 29431 raft_consensus.cc:491] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:21.642613 29431 raft_consensus.cc:513] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.643695 29431 leader_election.cc:290] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93c1cfb1ffbf4591bb2e193cc63d9e5e (127.26.229.195:39975), 9a5d0cfc3462438995fa4abeca06cd15 (127.26.229.194:38169)
I20250624 08:02:21.648051 29362 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541528f8db4946c2bbd99c2f7c66a681" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" is_pre_election: true
I20250624 08:02:21.648348 29362 raft_consensus.cc:2466] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 0.
I20250624 08:02:21.648669 29231 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541528f8db4946c2bbd99c2f7c66a681" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a5d0cfc3462438995fa4abeca06cd15" is_pre_election: true
I20250624 08:02:21.648876 29034 leader_election.cc:304] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c1cfb1ffbf4591bb2e193cc63d9e5e, f202001d5b444ce9926562c65505d5a5; no voters:
I20250624 08:02:21.648916 29231 raft_consensus.cc:2466] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 0.
I20250624 08:02:21.649101 29431 raft_consensus.cc:2802] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:21.649199 29431 raft_consensus.cc:491] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:21.649250 29431 raft_consensus.cc:3058] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:21.650103 29431 raft_consensus.cc:513] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.650437 29431 leader_election.cc:290] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 93c1cfb1ffbf4591bb2e193cc63d9e5e (127.26.229.195:39975), 9a5d0cfc3462438995fa4abeca06cd15 (127.26.229.194:38169)
I20250624 08:02:21.650768 29362 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541528f8db4946c2bbd99c2f7c66a681" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e"
I20250624 08:02:21.650768 29231 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541528f8db4946c2bbd99c2f7c66a681" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a5d0cfc3462438995fa4abeca06cd15"
I20250624 08:02:21.650902 29231 raft_consensus.cc:3058] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:21.650902 29362 raft_consensus.cc:3058] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:21.652034 29231 raft_consensus.cc:2466] T 541528f8db4946c2bbd99c2f7c66a681 P 9a5d0cfc3462438995fa4abeca06cd15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 1.
I20250624 08:02:21.652384 29036 leader_election.cc:304] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5d0cfc3462438995fa4abeca06cd15, f202001d5b444ce9926562c65505d5a5; no voters:
I20250624 08:02:21.652454 29362 raft_consensus.cc:2466] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 1.
I20250624 08:02:21.652578 29431 raft_consensus.cc:2802] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:21.652773 29431 raft_consensus.cc:695] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [term 1 LEADER]: Becoming Leader. State: Replica: f202001d5b444ce9926562c65505d5a5, State: Running, Role: LEADER
I20250624 08:02:21.653069 29431 consensus_queue.cc:237] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } }
I20250624 08:02:21.655839 28957 catalog_manager.cc:5582] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f202001d5b444ce9926562c65505d5a5 (127.26.229.193). New cstate: current_term: 1 leader_uuid: "f202001d5b444ce9926562c65505d5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } health_report { overall_health: UNKNOWN } } }
I20250624 08:02:21.663946 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:21.665189 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f202001d5b444ce9926562c65505d5a5 to finish bootstrapping
I20250624 08:02:21.669768 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9a5d0cfc3462438995fa4abeca06cd15 to finish bootstrapping
I20250624 08:02:21.673998 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 93c1cfb1ffbf4591bb2e193cc63d9e5e to finish bootstrapping
I20250624 08:02:21.679553 28957 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43998:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250624 08:02:21.679939 28957 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:02:21.684300 29080 tablet_service.cc:1468] Processing CreateTablet for tablet c8bbe1086fd349f3aeccbdbd0f1a7bdb (DEFAULT_TABLE table=TestAnotherTable [id=f27fa679ecb84cd18182498202d34f83]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 08:02:21.684340 29211 tablet_service.cc:1468] Processing CreateTablet for tablet c8bbe1086fd349f3aeccbdbd0f1a7bdb (DEFAULT_TABLE table=TestAnotherTable [id=f27fa679ecb84cd18182498202d34f83]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 08:02:21.684433 29342 tablet_service.cc:1468] Processing CreateTablet for tablet c8bbe1086fd349f3aeccbdbd0f1a7bdb (DEFAULT_TABLE table=TestAnotherTable [id=f27fa679ecb84cd18182498202d34f83]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 08:02:21.684624 29211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8bbe1086fd349f3aeccbdbd0f1a7bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.684621 29080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8bbe1086fd349f3aeccbdbd0f1a7bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.684676 29342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8bbe1086fd349f3aeccbdbd0f1a7bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:21.686430 29426 tablet_bootstrap.cc:492] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15: Bootstrap starting.
I20250624 08:02:21.686724 29425 tablet_bootstrap.cc:492] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Bootstrap starting.
I20250624 08:02:21.687199 29427 tablet_bootstrap.cc:492] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5: Bootstrap starting.
I20250624 08:02:21.687480 29426 tablet_bootstrap.cc:654] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.687680 29425 tablet_bootstrap.cc:654] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.688201 29427 tablet_bootstrap.cc:654] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:21.688731 29426 tablet_bootstrap.cc:492] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15: No bootstrap required, opened a new log
I20250624 08:02:21.688820 29426 ts_tablet_manager.cc:1397] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:21.688848 29425 tablet_bootstrap.cc:492] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e: No bootstrap required, opened a new log
I20250624 08:02:21.688920 29425 ts_tablet_manager.cc:1397] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:21.689280 29427 tablet_bootstrap.cc:492] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5: No bootstrap required, opened a new log
I20250624 08:02:21.689337 29427 ts_tablet_manager.cc:1397] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:21.689427 29426 raft_consensus.cc:357] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:21.689426 29425 raft_consensus.cc:357] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:21.689527 29426 raft_consensus.cc:383] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.689527 29425 raft_consensus.cc:383] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.689559 29426 raft_consensus.cc:738] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5d0cfc3462438995fa4abeca06cd15, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.689559 29425 raft_consensus.cc:738] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c1cfb1ffbf4591bb2e193cc63d9e5e, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.689687 29425 consensus_queue.cc:260] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:21.689687 29426 consensus_queue.cc:260] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:21.689745 29427 raft_consensus.cc:357] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:21.689843 29427 raft_consensus.cc:383] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:21.689879 29427 raft_consensus.cc:738] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f202001d5b444ce9926562c65505d5a5, State: Initialized, Role: FOLLOWER
I20250624 08:02:21.689941 29426 ts_tablet_manager.cc:1428] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:21.689936 29425 ts_tablet_manager.cc:1428] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:21.690033 29427 consensus_queue.cc:260] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:21.690292 29427 ts_tablet_manager.cc:1428] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250624 08:02:21.722651 29147 tablet.cc:2378] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 08:02:21.801394 29409 tablet.cc:2378] T 541528f8db4946c2bbd99c2f7c66a681 P 93c1cfb1ffbf4591bb2e193cc63d9e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:22.012794 29431 raft_consensus.cc:491] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:22.012929 29431 raft_consensus.cc:513] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:22.013367 29431 leader_election.cc:290] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a5d0cfc3462438995fa4abeca06cd15 (127.26.229.194:38169), 93c1cfb1ffbf4591bb2e193cc63d9e5e (127.26.229.195:39975)
I20250624 08:02:22.013839 29231 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8bbe1086fd349f3aeccbdbd0f1a7bdb" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a5d0cfc3462438995fa4abeca06cd15" is_pre_election: true
I20250624 08:02:22.013842 29362 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8bbe1086fd349f3aeccbdbd0f1a7bdb" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" is_pre_election: true
I20250624 08:02:22.013976 29362 raft_consensus.cc:2466] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 0.
I20250624 08:02:22.013984 29231 raft_consensus.cc:2466] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 0.
I20250624 08:02:22.014256 29034 leader_election.cc:304] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c1cfb1ffbf4591bb2e193cc63d9e5e, f202001d5b444ce9926562c65505d5a5; no voters:
I20250624 08:02:22.014420 29431 raft_consensus.cc:2802] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:22.014480 29431 raft_consensus.cc:491] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:22.014510 29431 raft_consensus.cc:3058] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:22.015409 29431 raft_consensus.cc:513] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:22.015719 29431 leader_election.cc:290] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a5d0cfc3462438995fa4abeca06cd15 (127.26.229.194:38169), 93c1cfb1ffbf4591bb2e193cc63d9e5e (127.26.229.195:39975)
I20250624 08:02:22.016057 29231 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8bbe1086fd349f3aeccbdbd0f1a7bdb" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a5d0cfc3462438995fa4abeca06cd15"
I20250624 08:02:22.016127 29362 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8bbe1086fd349f3aeccbdbd0f1a7bdb" candidate_uuid: "f202001d5b444ce9926562c65505d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e"
I20250624 08:02:22.016181 29231 raft_consensus.cc:3058] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:22.016232 29362 raft_consensus.cc:3058] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:22.016876 29231 raft_consensus.cc:2466] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 9a5d0cfc3462438995fa4abeca06cd15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 1.
I20250624 08:02:22.017131 29362 raft_consensus.cc:2466] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P 93c1cfb1ffbf4591bb2e193cc63d9e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f202001d5b444ce9926562c65505d5a5 in term 1.
I20250624 08:02:22.017180 29036 leader_election.cc:304] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5d0cfc3462438995fa4abeca06cd15, f202001d5b444ce9926562c65505d5a5; no voters:
I20250624 08:02:22.017354 29431 raft_consensus.cc:2802] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:22.017448 29431 raft_consensus.cc:695] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [term 1 LEADER]: Becoming Leader. State: Replica: f202001d5b444ce9926562c65505d5a5, State: Running, Role: LEADER
I20250624 08:02:22.017792 29431 consensus_queue.cc:237] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } }
I20250624 08:02:22.019408 28958 catalog_manager.cc:5582] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f202001d5b444ce9926562c65505d5a5 (127.26.229.193). New cstate: current_term: 1 leader_uuid: "f202001d5b444ce9926562c65505d5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f202001d5b444ce9926562c65505d5a5" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 45471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 } health_report { overall_health: UNKNOWN } } }
W20250624 08:02:22.148257 29442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:22.148514 29442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250624 08:02:22.151597 29431 consensus_queue.cc:1035] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 08:02:22.152148 29442 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 08:02:22.157068 29431 consensus_queue.cc:1035] T 541528f8db4946c2bbd99c2f7c66a681 P f202001d5b444ce9926562c65505d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 08:02:22.288825 29458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:22.289023 29458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:22.292052 29458 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 08:02:22.428169 29468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:22.428362 29468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:22.431105 29468 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 08:02:22.470849 29435 consensus_queue.cc:1035] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5d0cfc3462438995fa4abeca06cd15" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:22.474542 29435 consensus_queue.cc:1035] T c8bbe1086fd349f3aeccbdbd0f1a7bdb P f202001d5b444ce9926562c65505d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c1cfb1ffbf4591bb2e193cc63d9e5e" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 08:02:22.568154 29478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:22.568387 29478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:22.571786 29478 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 08:02:22.604964 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29018
I20250624 08:02:22.610842 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29149
I20250624 08:02:22.616381 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29280
I20250624 08:02:22.621552 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 28927
2025-06-24T08:02:22Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (1798 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250624 08:02:22.643086 27543 test_util.cc:276] Using random seed: -1081258102
I20250624 08:02:22.643797 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:22.643836 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:22.643847 27543 ts_itest-base.cc:117] 3 tablet servers
I20250624 08:02:22.643857 27543 ts_itest-base.cc:118] 3 replicas per TS
I20250624 08:02:22.643867 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:22Z Disabled control of system clock
I20250624 08:02:22.651422 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:42157
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:42157
--superuser_acl=no-such-user with env {}
W20250624 08:02:22.747951 29498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:22.748137 29498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:22.748214 29498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:22.751060 29498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:22.751112 29498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:22.751144 29498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:22.751184 29498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:22.754469 29498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:42157
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.29498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:22.755445 29498 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:22.756477 29498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:22.762821 29503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:22.762842 29506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:22.762954 29498 server_base.cc:1048] running on GCE node
W20250624 08:02:22.762821 29504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:22.763514 29498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:22.764314 29498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:22.765444 29498 hybrid_clock.cc:648] HybridClock initialized: now 1750752142765429 us; error 29 us; skew 500 ppm
I20250624 08:02:22.771287 29498 webserver.cc:469] Webserver started at http://127.26.229.254:39089/ using document root <none> and password file <none>
I20250624 08:02:22.771811 29498 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:22.771862 29498 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:22.772017 29498 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:22.773489 29498 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "60085c8e03fa4ef6aac78846ad14e695"
format_stamp: "Formatted at 2025-06-24 08:02:22 on dist-test-slave-4j8z"
I20250624 08:02:22.773892 29498 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "60085c8e03fa4ef6aac78846ad14e695"
format_stamp: "Formatted at 2025-06-24 08:02:22 on dist-test-slave-4j8z"
I20250624 08:02:22.776984 29498 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250624 08:02:22.779033 29512 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:22.779945 29498 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250624 08:02:22.780035 29498 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "60085c8e03fa4ef6aac78846ad14e695"
format_stamp: "Formatted at 2025-06-24 08:02:22 on dist-test-slave-4j8z"
I20250624 08:02:22.780110 29498 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:22.794728 29498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:22.795372 29498 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:22.795511 29498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:22.802412 29564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:42157 every 8 connection(s)
I20250624 08:02:22.802410 29498 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:42157
I20250624 08:02:22.803463 29498 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:22.805876 29565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:22.805920 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29498
I20250624 08:02:22.806015 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:22.811237 29565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695: Bootstrap starting.
I20250624 08:02:22.813335 29565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:22.813936 29565 log.cc:826] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:22.815380 29565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695: No bootstrap required, opened a new log
I20250624 08:02:22.817384 29565 raft_consensus.cc:357] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60085c8e03fa4ef6aac78846ad14e695" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 42157 } }
I20250624 08:02:22.817548 29565 raft_consensus.cc:383] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:22.817621 29565 raft_consensus.cc:738] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60085c8e03fa4ef6aac78846ad14e695, State: Initialized, Role: FOLLOWER
I20250624 08:02:22.817968 29565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60085c8e03fa4ef6aac78846ad14e695" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 42157 } }
I20250624 08:02:22.818090 29565 raft_consensus.cc:397] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:22.818151 29565 raft_consensus.cc:491] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:22.818256 29565 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:22.818961 29565 raft_consensus.cc:513] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60085c8e03fa4ef6aac78846ad14e695" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 42157 } }
I20250624 08:02:22.819371 29565 leader_election.cc:304] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60085c8e03fa4ef6aac78846ad14e695; no voters:
I20250624 08:02:22.819648 29565 leader_election.cc:290] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:22.819734 29570 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:22.819945 29570 raft_consensus.cc:695] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [term 1 LEADER]: Becoming Leader. State: Replica: 60085c8e03fa4ef6aac78846ad14e695, State: Running, Role: LEADER
I20250624 08:02:22.820263 29570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60085c8e03fa4ef6aac78846ad14e695" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 42157 } }
I20250624 08:02:22.820581 29565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:22.821707 29571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60085c8e03fa4ef6aac78846ad14e695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60085c8e03fa4ef6aac78846ad14e695" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 42157 } } }
I20250624 08:02:22.821837 29571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:22.822173 29579 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:22.822732 29572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60085c8e03fa4ef6aac78846ad14e695. Latest consensus state: current_term: 1 leader_uuid: "60085c8e03fa4ef6aac78846ad14e695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60085c8e03fa4ef6aac78846ad14e695" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 42157 } } }
I20250624 08:02:22.822885 29572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:22.824918 29579 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:22.829496 29579 catalog_manager.cc:1349] Generated new cluster ID: fe52b2a1ef9d4f7b9fcef9bbd4d733f5
I20250624 08:02:22.829571 29579 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:22.835289 29579 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:22.836063 29579 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:22.843909 29579 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 60085c8e03fa4ef6aac78846ad14e695: Generated new TSK 0
I20250624 08:02:22.844468 29579 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:22.850235 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:42157
--builtin_ntp_servers=127.26.229.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 08:02:22.948864 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:22.949079 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:22.949120 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:22.952229 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:22.952325 29589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:22.955907 29589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:42157
--never_fsync=true
--heap_profile_path=/tmp/kudu.29589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:22.956926 29589 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:22.958050 29589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:22.965361 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:22.965369 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:22.965734 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:22.966125 29589 server_base.cc:1048] running on GCE node
I20250624 08:02:22.966497 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:22.967105 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:22.968278 29589 hybrid_clock.cc:648] HybridClock initialized: now 1750752142968248 us; error 37 us; skew 500 ppm
I20250624 08:02:22.970449 29589 webserver.cc:469] Webserver started at http://127.26.229.193:38029/ using document root <none> and password file <none>
I20250624 08:02:22.971136 29589 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:22.971211 29589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:22.971426 29589 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:22.973323 29589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2c45b4ea06f84f62926394052ba605cb"
format_stamp: "Formatted at 2025-06-24 08:02:22 on dist-test-slave-4j8z"
I20250624 08:02:22.973898 29589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2c45b4ea06f84f62926394052ba605cb"
format_stamp: "Formatted at 2025-06-24 08:02:22 on dist-test-slave-4j8z"
I20250624 08:02:22.978299 29589 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250624 08:02:22.981163 29603 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:22.982395 29589 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:22.982491 29589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c45b4ea06f84f62926394052ba605cb"
format_stamp: "Formatted at 2025-06-24 08:02:22 on dist-test-slave-4j8z"
I20250624 08:02:22.982632 29589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:22.998878 29589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:22.999486 29589 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:22.999666 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:23.000339 29589 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:23.001503 29589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:23.001567 29589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.001626 29589 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:23.001701 29589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.014906 29589 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:33345
I20250624 08:02:23.014927 29716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:33345 every 8 connection(s)
I20250624 08:02:23.016330 29589 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:23.025687 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29589
I20250624 08:02:23.025786 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:23.025817 29717 heartbeater.cc:344] Connected to a master server at 127.26.229.254:42157
I20250624 08:02:23.026013 29717 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:23.026582 29717 heartbeater.cc:507] Master 127.26.229.254:42157 requested a full tablet report, sending...
I20250624 08:02:23.027845 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:42157
--builtin_ntp_servers=127.26.229.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:23.027987 29529 ts_manager.cc:194] Registered new tserver with Master: 2c45b4ea06f84f62926394052ba605cb (127.26.229.193:33345)
I20250624 08:02:23.029295 29529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:56379
W20250624 08:02:23.129979 29720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:23.130174 29720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:23.130218 29720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:23.133379 29720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:23.133473 29720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:23.136914 29720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:42157
--never_fsync=true
--heap_profile_path=/tmp/kudu.29720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:23.137879 29720 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:23.138922 29720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:23.146307 29726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:23.146319 29728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:23.146270 29725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:23.146440 29720 server_base.cc:1048] running on GCE node
I20250624 08:02:23.146847 29720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:23.147341 29720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:23.148492 29720 hybrid_clock.cc:648] HybridClock initialized: now 1750752143148470 us; error 36 us; skew 500 ppm
I20250624 08:02:23.150071 29720 webserver.cc:469] Webserver started at http://127.26.229.194:46749/ using document root <none> and password file <none>
I20250624 08:02:23.150565 29720 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:23.150614 29720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:23.150763 29720 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:23.152285 29720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2c71d1c23d8743a88a4ae32fb6490bf0"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.152736 29720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2c71d1c23d8743a88a4ae32fb6490bf0"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.155558 29720 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250624 08:02:23.157763 29734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.158627 29720 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:23.158727 29720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2c71d1c23d8743a88a4ae32fb6490bf0"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.158823 29720 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:23.176733 29720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:23.177299 29720 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:23.177456 29720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:23.177966 29720 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:23.178825 29720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:23.178880 29720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.178946 29720 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:23.178974 29720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.187526 29720 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:42599
I20250624 08:02:23.187551 29847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:42599 every 8 connection(s)
I20250624 08:02:23.188530 29720 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:23.193169 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29720
I20250624 08:02:23.193288 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 08:02:23.195437 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:42157
--builtin_ntp_servers=127.26.229.212:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:23.197398 29848 heartbeater.cc:344] Connected to a master server at 127.26.229.254:42157
I20250624 08:02:23.197652 29848 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:23.198155 29848 heartbeater.cc:507] Master 127.26.229.254:42157 requested a full tablet report, sending...
I20250624 08:02:23.199159 29529 ts_manager.cc:194] Registered new tserver with Master: 2c71d1c23d8743a88a4ae32fb6490bf0 (127.26.229.194:42599)
I20250624 08:02:23.199906 29529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:43945
W20250624 08:02:23.292270 29851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:23.292454 29851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:23.292495 29851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:23.295365 29851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:23.295455 29851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:23.299002 29851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:42157
--never_fsync=true
--heap_profile_path=/tmp/kudu.29851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:23.299975 29851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:23.301088 29851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:23.308544 29859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:23.308679 29856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:23.308715 29857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:23.308928 29851 server_base.cc:1048] running on GCE node
I20250624 08:02:23.309295 29851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:23.309813 29851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:23.310984 29851 hybrid_clock.cc:648] HybridClock initialized: now 1750752143310967 us; error 27 us; skew 500 ppm
I20250624 08:02:23.312681 29851 webserver.cc:469] Webserver started at http://127.26.229.195:34997/ using document root <none> and password file <none>
I20250624 08:02:23.313269 29851 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:23.313323 29851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:23.313496 29851 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:23.315014 29851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "32aa254f09bf41ad9fa5eea22a2a48c0"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.315425 29851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "32aa254f09bf41ad9fa5eea22a2a48c0"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.318361 29851 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:23.320428 29865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.321390 29851 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250624 08:02:23.321484 29851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "32aa254f09bf41ad9fa5eea22a2a48c0"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.321558 29851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:23.340158 29851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:23.340754 29851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:23.340916 29851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:23.341432 29851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:23.342262 29851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:23.342310 29851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.342366 29851 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:23.342392 29851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.350800 29851 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:45075
I20250624 08:02:23.350836 29978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:45075 every 8 connection(s)
I20250624 08:02:23.351697 29851 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:23.359692 29979 heartbeater.cc:344] Connected to a master server at 127.26.229.254:42157
I20250624 08:02:23.359925 29979 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:23.360437 29979 heartbeater.cc:507] Master 127.26.229.254:42157 requested a full tablet report, sending...
I20250624 08:02:23.360745 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 29851
I20250624 08:02:23.360838 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 08:02:23.361416 29529 ts_manager.cc:194] Registered new tserver with Master: 32aa254f09bf41ad9fa5eea22a2a48c0 (127.26.229.195:45075)
I20250624 08:02:23.362001 29529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:44051
I20250624 08:02:23.363363 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:23.370690 29529 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52076:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 08:02:23.373085 29529 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:02:23.388312 29782 tablet_service.cc:1468] Processing CreateTablet for tablet b289b23c779c48c1ad335eea0b79c201 (DEFAULT_TABLE table=TestTable [id=00d02cdff4684b9bbedea9cc9411529e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:23.388306 29651 tablet_service.cc:1468] Processing CreateTablet for tablet b289b23c779c48c1ad335eea0b79c201 (DEFAULT_TABLE table=TestTable [id=00d02cdff4684b9bbedea9cc9411529e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:23.388306 29913 tablet_service.cc:1468] Processing CreateTablet for tablet b289b23c779c48c1ad335eea0b79c201 (DEFAULT_TABLE table=TestTable [id=00d02cdff4684b9bbedea9cc9411529e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:23.389178 29782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b289b23c779c48c1ad335eea0b79c201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:23.389191 29913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b289b23c779c48c1ad335eea0b79c201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:23.389192 29651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b289b23c779c48c1ad335eea0b79c201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:23.394042 29996 tablet_bootstrap.cc:492] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: Bootstrap starting.
I20250624 08:02:23.395056 29998 tablet_bootstrap.cc:492] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0: Bootstrap starting.
I20250624 08:02:23.396154 29997 tablet_bootstrap.cc:492] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: Bootstrap starting.
I20250624 08:02:23.396485 29996 tablet_bootstrap.cc:654] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:23.396970 29998 tablet_bootstrap.cc:654] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:23.397330 29996 log.cc:826] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:23.397707 29998 log.cc:826] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:23.398077 29997 tablet_bootstrap.cc:654] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:23.398818 29997 log.cc:826] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:23.399143 29996 tablet_bootstrap.cc:492] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: No bootstrap required, opened a new log
I20250624 08:02:23.399178 29998 tablet_bootstrap.cc:492] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0: No bootstrap required, opened a new log
I20250624 08:02:23.399345 29996 ts_tablet_manager.cc:1397] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250624 08:02:23.399362 29998 ts_tablet_manager.cc:1397] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250624 08:02:23.400432 29997 tablet_bootstrap.cc:492] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: No bootstrap required, opened a new log
I20250624 08:02:23.400596 29997 ts_tablet_manager.cc:1397] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250624 08:02:23.401746 29998 raft_consensus.cc:357] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.401962 29998 raft_consensus.cc:383] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:23.402024 29998 raft_consensus.cc:738] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa254f09bf41ad9fa5eea22a2a48c0, State: Initialized, Role: FOLLOWER
I20250624 08:02:23.402293 29996 raft_consensus.cc:357] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.402489 29996 raft_consensus.cc:383] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:23.402446 29998 consensus_queue.cc:260] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.402544 29996 raft_consensus.cc:738] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c71d1c23d8743a88a4ae32fb6490bf0, State: Initialized, Role: FOLLOWER
I20250624 08:02:23.402988 29996 consensus_queue.cc:260] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.403072 29979 heartbeater.cc:499] Master 127.26.229.254:42157 was elected leader, sending a full tablet report...
I20250624 08:02:23.403295 29998 ts_tablet_manager.cc:1428] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 08:02:23.403421 29997 raft_consensus.cc:357] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.403592 29997 raft_consensus.cc:383] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:23.403646 29997 raft_consensus.cc:738] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c45b4ea06f84f62926394052ba605cb, State: Initialized, Role: FOLLOWER
I20250624 08:02:23.403761 29996 ts_tablet_manager.cc:1428] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 08:02:23.404002 29997 consensus_queue.cc:260] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.404083 29848 heartbeater.cc:499] Master 127.26.229.254:42157 was elected leader, sending a full tablet report...
I20250624 08:02:23.404616 29717 heartbeater.cc:499] Master 127.26.229.254:42157 was elected leader, sending a full tablet report...
I20250624 08:02:23.404758 29997 ts_tablet_manager.cc:1428] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:23.406543 30004 raft_consensus.cc:491] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:23.406673 30004 raft_consensus.cc:513] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.407781 30004 leader_election.cc:290] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32aa254f09bf41ad9fa5eea22a2a48c0 (127.26.229.195:45075), 2c71d1c23d8743a88a4ae32fb6490bf0 (127.26.229.194:42599)
I20250624 08:02:23.411643 29933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b289b23c779c48c1ad335eea0b79c201" candidate_uuid: "2c45b4ea06f84f62926394052ba605cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" is_pre_election: true
I20250624 08:02:23.411695 29801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b289b23c779c48c1ad335eea0b79c201" candidate_uuid: "2c45b4ea06f84f62926394052ba605cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" is_pre_election: true
I20250624 08:02:23.411859 29933 raft_consensus.cc:2466] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c45b4ea06f84f62926394052ba605cb in term 0.
I20250624 08:02:23.411890 29801 raft_consensus.cc:2466] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c45b4ea06f84f62926394052ba605cb in term 0.
I20250624 08:02:23.412262 29606 leader_election.cc:304] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c45b4ea06f84f62926394052ba605cb, 2c71d1c23d8743a88a4ae32fb6490bf0; no voters:
I20250624 08:02:23.412478 30004 raft_consensus.cc:2802] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:23.412544 30004 raft_consensus.cc:491] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:23.412577 30004 raft_consensus.cc:3058] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:23.413385 30004 raft_consensus.cc:513] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.413691 30004 leader_election.cc:290] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [CANDIDATE]: Term 1 election: Requested vote from peers 32aa254f09bf41ad9fa5eea22a2a48c0 (127.26.229.195:45075), 2c71d1c23d8743a88a4ae32fb6490bf0 (127.26.229.194:42599)
I20250624 08:02:23.414002 29933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b289b23c779c48c1ad335eea0b79c201" candidate_uuid: "2c45b4ea06f84f62926394052ba605cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0"
I20250624 08:02:23.414073 29801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b289b23c779c48c1ad335eea0b79c201" candidate_uuid: "2c45b4ea06f84f62926394052ba605cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0"
I20250624 08:02:23.414137 29933 raft_consensus.cc:3058] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:23.414163 29801 raft_consensus.cc:3058] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:23.415711 29933 raft_consensus.cc:2466] T b289b23c779c48c1ad335eea0b79c201 P 32aa254f09bf41ad9fa5eea22a2a48c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c45b4ea06f84f62926394052ba605cb in term 1.
I20250624 08:02:23.415711 29801 raft_consensus.cc:2466] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c45b4ea06f84f62926394052ba605cb in term 1.
I20250624 08:02:23.416013 29606 leader_election.cc:304] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c45b4ea06f84f62926394052ba605cb, 32aa254f09bf41ad9fa5eea22a2a48c0; no voters:
I20250624 08:02:23.416191 30004 raft_consensus.cc:2802] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:23.416419 30004 raft_consensus.cc:695] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [term 1 LEADER]: Becoming Leader. State: Replica: 2c45b4ea06f84f62926394052ba605cb, State: Running, Role: LEADER
I20250624 08:02:23.416971 30004 consensus_queue.cc:237] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } }
I20250624 08:02:23.419737 29527 catalog_manager.cc:5582] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c45b4ea06f84f62926394052ba605cb (127.26.229.193). New cstate: current_term: 1 leader_uuid: "2c45b4ea06f84f62926394052ba605cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aa254f09bf41ad9fa5eea22a2a48c0" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c71d1c23d8743a88a4ae32fb6490bf0" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c45b4ea06f84f62926394052ba605cb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33345 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:23.430744 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:23.432152 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2c45b4ea06f84f62926394052ba605cb to finish bootstrapping
I20250624 08:02:23.437220 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2c71d1c23d8743a88a4ae32fb6490bf0 to finish bootstrapping
W20250624 08:02:23.439368 29849 tablet.cc:2378] T b289b23c779c48c1ad335eea0b79c201 P 2c71d1c23d8743a88a4ae32fb6490bf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:23.441658 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 32aa254f09bf41ad9fa5eea22a2a48c0 to finish bootstrapping
W20250624 08:02:23.517590 29718 tablet.cc:2378] T b289b23c779c48c1ad335eea0b79c201 P 2c45b4ea06f84f62926394052ba605cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 08:02:23.565207 29527 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:52086
I20250624 08:02:23.572186 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29589
I20250624 08:02:23.578064 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29720
I20250624 08:02:23.583297 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29851
I20250624 08:02:23.588461 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 29498
2025-06-24T08:02:23Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (967 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250624 08:02:23.610141 27543 test_util.cc:276] Using random seed: -1080291048
I20250624 08:02:23.610878 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:23.610925 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:23.610946 27543 ts_itest-base.cc:117] 3 tablet servers
I20250624 08:02:23.610965 27543 ts_itest-base.cc:118] 3 replicas per TS
I20250624 08:02:23.610983 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:23Z Disabled control of system clock
I20250624 08:02:23.618368 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:39173
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:39173 with env {}
W20250624 08:02:23.715332 30031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:23.715514 30031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:23.715550 30031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:23.718461 30031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:23.718508 30031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:23.718525 30031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:23.718542 30031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:23.721998 30031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:39173
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.30031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:23.722958 30031 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:23.723958 30031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:23.730175 30039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:23.730181 30036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:23.730286 30031 server_base.cc:1048] running on GCE node
W20250624 08:02:23.730191 30037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:23.730892 30031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:23.731685 30031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:23.732827 30031 hybrid_clock.cc:648] HybridClock initialized: now 1750752143732804 us; error 37 us; skew 500 ppm
I20250624 08:02:23.734462 30031 webserver.cc:469] Webserver started at http://127.26.229.254:34043/ using document root <none> and password file <none>
I20250624 08:02:23.735004 30031 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:23.735066 30031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:23.735244 30031 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:23.736683 30031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "997da62dfb914b9bb45a72d56c09d8f6"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.737123 30031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "997da62dfb914b9bb45a72d56c09d8f6"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.740291 30031 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:23.742244 30045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.743222 30031 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:23.743335 30031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "997da62dfb914b9bb45a72d56c09d8f6"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.743415 30031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:23.756919 30031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:23.757503 30031 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:23.757673 30031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:23.764077 30031 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:39173
I20250624 08:02:23.764082 30097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:39173 every 8 connection(s)
I20250624 08:02:23.764958 30031 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:23.767210 30098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:23.771616 30098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap starting.
I20250624 08:02:23.772694 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30031
I20250624 08:02:23.772792 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:23.773839 30098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:23.774616 30098 log.cc:826] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:23.776094 30098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: No bootstrap required, opened a new log
I20250624 08:02:23.778308 30098 raft_consensus.cc:357] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:23.778484 30098 raft_consensus.cc:383] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:23.778558 30098 raft_consensus.cc:738] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997da62dfb914b9bb45a72d56c09d8f6, State: Initialized, Role: FOLLOWER
I20250624 08:02:23.779047 30098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:23.779172 30098 raft_consensus.cc:397] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:23.779235 30098 raft_consensus.cc:491] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:23.779454 30098 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:23.780331 30098 raft_consensus.cc:513] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:23.780619 30098 leader_election.cc:304] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997da62dfb914b9bb45a72d56c09d8f6; no voters:
I20250624 08:02:23.780874 30098 leader_election.cc:290] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:23.780948 30103 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:23.781108 30103 raft_consensus.cc:695] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 1 LEADER]: Becoming Leader. State: Replica: 997da62dfb914b9bb45a72d56c09d8f6, State: Running, Role: LEADER
I20250624 08:02:23.781397 30103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:23.781934 30098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:23.782444 30103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997da62dfb914b9bb45a72d56c09d8f6. Latest consensus state: current_term: 1 leader_uuid: "997da62dfb914b9bb45a72d56c09d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } } }
I20250624 08:02:23.782563 30103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:23.782936 30104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "997da62dfb914b9bb45a72d56c09d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } } }
I20250624 08:02:23.783136 30104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:23.782941 30107 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:23.785734 30107 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:23.790015 30107 catalog_manager.cc:1349] Generated new cluster ID: ea9f3b2913994e4194bc2a926e42481f
I20250624 08:02:23.790084 30107 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:23.813227 30107 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:23.814393 30107 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:23.827997 30107 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Generated new TSK 0
I20250624 08:02:23.828815 30107 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:23.835889 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:39173
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 08:02:23.937443 30122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:23.937657 30122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:23.937727 30122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:23.940847 30122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:23.940984 30122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:23.944588 30122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:39173
--never_fsync=true
--heap_profile_path=/tmp/kudu.30122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:23.945592 30122 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:23.946648 30122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:23.953953 30128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:23.953907 30127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:23.954056 30122 server_base.cc:1048] running on GCE node
W20250624 08:02:23.953907 30130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:23.954473 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:23.955027 30122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:23.956199 30122 hybrid_clock.cc:648] HybridClock initialized: now 1750752143956179 us; error 33 us; skew 500 ppm
I20250624 08:02:23.960731 30122 webserver.cc:469] Webserver started at http://127.26.229.193:42361/ using document root <none> and password file <none>
I20250624 08:02:23.961302 30122 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:23.961359 30122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:23.961522 30122 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:23.963128 30122 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "50a1ce8727584f8fbd564646831ac49d"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.963567 30122 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "50a1ce8727584f8fbd564646831ac49d"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.966698 30122 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250624 08:02:23.968750 30136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.969755 30122 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:23.969873 30122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "50a1ce8727584f8fbd564646831ac49d"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:23.969962 30122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:23.981061 30122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:23.981603 30122 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:23.981757 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:23.982236 30122 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:23.983150 30122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:23.983201 30122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.983254 30122 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:23.983287 30122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:23.991708 30122 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:39177
I20250624 08:02:23.991745 30249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:39177 every 8 connection(s)
I20250624 08:02:23.992556 30122 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:24.000651 30250 heartbeater.cc:344] Connected to a master server at 127.26.229.254:39173
I20250624 08:02:24.000887 30250 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:24.001310 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30122
I20250624 08:02:24.001368 30250 heartbeater.cc:507] Master 127.26.229.254:39173 requested a full tablet report, sending...
I20250624 08:02:24.001423 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:24.002707 30062 ts_manager.cc:194] Registered new tserver with Master: 50a1ce8727584f8fbd564646831ac49d (127.26.229.193:39177)
I20250624 08:02:24.003412 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:39173
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:24.003891 30062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:37543
W20250624 08:02:24.099956 30253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:24.100160 30253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:24.100201 30253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:24.103299 30253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:24.103423 30253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:24.106786 30253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:39173
--never_fsync=true
--heap_profile_path=/tmp/kudu.30253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:24.107748 30253 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:24.108737 30253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:24.115713 30261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:24.115726 30258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:24.115727 30259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:24.115847 30253 server_base.cc:1048] running on GCE node
I20250624 08:02:24.116312 30253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:24.116803 30253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:24.117935 30253 hybrid_clock.cc:648] HybridClock initialized: now 1750752144117927 us; error 26 us; skew 500 ppm
I20250624 08:02:24.119587 30253 webserver.cc:469] Webserver started at http://127.26.229.194:45915/ using document root <none> and password file <none>
I20250624 08:02:24.120083 30253 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:24.120136 30253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:24.120311 30253 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:24.121765 30253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "856af570ad924de3acb21ad0f7f9a490"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:24.122195 30253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "856af570ad924de3acb21ad0f7f9a490"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:24.125274 30253 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:24.127379 30267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:24.128420 30253 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:24.128520 30253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "856af570ad924de3acb21ad0f7f9a490"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:24.128600 30253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:24.146678 30253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:24.147351 30253 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:24.147545 30253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:24.148073 30253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:24.148950 30253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:24.149000 30253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:24.149073 30253 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:24.149102 30253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:24.157697 30253 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:44177
I20250624 08:02:24.157724 30380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:44177 every 8 connection(s)
I20250624 08:02:24.158600 30253 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:24.167197 30381 heartbeater.cc:344] Connected to a master server at 127.26.229.254:39173
I20250624 08:02:24.167443 30381 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:24.167954 30381 heartbeater.cc:507] Master 127.26.229.254:39173 requested a full tablet report, sending...
I20250624 08:02:24.168833 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30253
I20250624 08:02:24.168941 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 08:02:24.169013 30062 ts_manager.cc:194] Registered new tserver with Master: 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194:44177)
I20250624 08:02:24.169665 30062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:52871
I20250624 08:02:24.170997 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:39173
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 08:02:24.273478 30384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:24.273702 30384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:24.273775 30384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:24.276998 30384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:24.277143 30384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:24.281865 30384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:39173
--never_fsync=true
--heap_profile_path=/tmp/kudu.30384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:24.283004 30384 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:24.284133 30384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:24.291255 30389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:24.291392 30390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:24.291271 30392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:24.291857 30384 server_base.cc:1048] running on GCE node
I20250624 08:02:24.292263 30384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:24.292835 30384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:24.294008 30384 hybrid_clock.cc:648] HybridClock initialized: now 1750752144293978 us; error 41 us; skew 500 ppm
I20250624 08:02:24.295857 30384 webserver.cc:469] Webserver started at http://127.26.229.195:43379/ using document root <none> and password file <none>
I20250624 08:02:24.296429 30384 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:24.296492 30384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:24.296675 30384 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:24.298184 30384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7aadeff39afb4ba6b4e0d692791b53b2"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:24.298674 30384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7aadeff39afb4ba6b4e0d692791b53b2"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:24.301915 30384 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:24.304100 30398 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:24.305059 30384 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:24.305162 30384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7aadeff39afb4ba6b4e0d692791b53b2"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:24.305260 30384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:24.322889 30384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:24.323649 30384 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:24.323843 30384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:24.324438 30384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:24.325373 30384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:24.325441 30384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:24.325518 30384 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:24.325544 30384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:24.334589 30384 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:42011
I20250624 08:02:24.334606 30511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:42011 every 8 connection(s)
I20250624 08:02:24.335642 30384 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:24.336099 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30384
I20250624 08:02:24.336200 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 08:02:24.344352 30512 heartbeater.cc:344] Connected to a master server at 127.26.229.254:39173
I20250624 08:02:24.344587 30512 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:24.345104 30512 heartbeater.cc:507] Master 127.26.229.254:39173 requested a full tablet report, sending...
I20250624 08:02:24.346048 30061 ts_manager.cc:194] Registered new tserver with Master: 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195:42011)
I20250624 08:02:24.346597 30061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:46509
I20250624 08:02:24.350059 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:24.358973 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:24.359056 27543 test_util.cc:276] Using random seed: -1079542127
I20250624 08:02:24.368268 30061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53256:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 08:02:24.382679 30184 tablet_service.cc:1468] Processing CreateTablet for tablet 855d2623a4d6420a9de9a38744eed166 (DEFAULT_TABLE table=TestTable [id=fdec9415a81d458c9e6e210ffb833d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:24.383664 30184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855d2623a4d6420a9de9a38744eed166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:24.388978 30530 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap starting.
I20250624 08:02:24.391170 30530 tablet_bootstrap.cc:654] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:24.391872 30530 log.cc:826] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:24.393533 30530 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: No bootstrap required, opened a new log
I20250624 08:02:24.393687 30530 ts_tablet_manager.cc:1397] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250624 08:02:24.395931 30530 raft_consensus.cc:357] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:24.396107 30530 raft_consensus.cc:383] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:24.396143 30530 raft_consensus.cc:738] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a1ce8727584f8fbd564646831ac49d, State: Initialized, Role: FOLLOWER
I20250624 08:02:24.396472 30530 consensus_queue.cc:260] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:24.396567 30530 raft_consensus.cc:397] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:24.396610 30530 raft_consensus.cc:491] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:24.396685 30530 raft_consensus.cc:3058] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:24.397401 30530 raft_consensus.cc:513] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:24.397720 30530 leader_election.cc:304] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50a1ce8727584f8fbd564646831ac49d; no voters:
I20250624 08:02:24.397950 30530 leader_election.cc:290] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:24.398072 30532 raft_consensus.cc:2802] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:24.398309 30532 raft_consensus.cc:695] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 LEADER]: Becoming Leader. State: Replica: 50a1ce8727584f8fbd564646831ac49d, State: Running, Role: LEADER
I20250624 08:02:24.398600 30532 consensus_queue.cc:237] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:24.398802 30530 ts_tablet_manager.cc:1428] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20250624 08:02:24.398980 30250 heartbeater.cc:499] Master 127.26.229.254:39173 was elected leader, sending a full tablet report...
I20250624 08:02:24.402122 30061 catalog_manager.cc:5582] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 50a1ce8727584f8fbd564646831ac49d (127.26.229.193). New cstate: current_term: 1 leader_uuid: "50a1ce8727584f8fbd564646831ac49d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:24.444427 27543 test_util.cc:276] Using random seed: -1079456766
I20250624 08:02:24.451848 30061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53266:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 08:02:24.462819 30446 tablet_service.cc:1468] Processing CreateTablet for tablet 9bfc26ce4f374a048595bfce459771d4 (DEFAULT_TABLE table=TestTable1 [id=ceb8d3d6d2cf45b4adbd5032bbeb9e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:24.463711 30446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfc26ce4f374a048595bfce459771d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:24.469019 30552 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap starting.
I20250624 08:02:24.471201 30552 tablet_bootstrap.cc:654] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:24.471907 30552 log.cc:826] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:24.473405 30552 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: No bootstrap required, opened a new log
I20250624 08:02:24.473579 30552 ts_tablet_manager.cc:1397] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250624 08:02:24.476039 30552 raft_consensus.cc:357] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:24.476218 30552 raft_consensus.cc:383] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:24.476267 30552 raft_consensus.cc:738] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadeff39afb4ba6b4e0d692791b53b2, State: Initialized, Role: FOLLOWER
I20250624 08:02:24.476680 30552 consensus_queue.cc:260] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:24.476795 30552 raft_consensus.cc:397] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:24.476861 30552 raft_consensus.cc:491] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:24.476959 30552 raft_consensus.cc:3058] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:24.477749 30552 raft_consensus.cc:513] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:24.478063 30552 leader_election.cc:304] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aadeff39afb4ba6b4e0d692791b53b2; no voters:
I20250624 08:02:24.478340 30552 leader_election.cc:290] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:24.478509 30554 raft_consensus.cc:2802] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:24.478708 30554 raft_consensus.cc:695] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 LEADER]: Becoming Leader. State: Replica: 7aadeff39afb4ba6b4e0d692791b53b2, State: Running, Role: LEADER
I20250624 08:02:24.478945 30512 heartbeater.cc:499] Master 127.26.229.254:39173 was elected leader, sending a full tablet report...
I20250624 08:02:24.479033 30554 consensus_queue.cc:237] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:24.479167 30552 ts_tablet_manager.cc:1428] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250624 08:02:24.482002 30061 catalog_manager.cc:5582] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195). New cstate: current_term: 1 leader_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:24.523794 27543 test_util.cc:276] Using random seed: -1079377402
I20250624 08:02:24.532087 30062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53274:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 08:02:24.542738 30315 tablet_service.cc:1468] Processing CreateTablet for tablet 615ee19f6e234df2b717bb61609b1b44 (DEFAULT_TABLE table=TestTable2 [id=0f108a40c3d44c298e1291ded5ada962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:24.543663 30315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615ee19f6e234df2b717bb61609b1b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:24.548405 30574 tablet_bootstrap.cc:492] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap starting.
I20250624 08:02:24.550531 30574 tablet_bootstrap.cc:654] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:24.551306 30574 log.cc:826] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:24.552829 30574 tablet_bootstrap.cc:492] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: No bootstrap required, opened a new log
I20250624 08:02:24.553012 30574 ts_tablet_manager.cc:1397] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250624 08:02:24.555716 30574 raft_consensus.cc:357] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:24.555892 30574 raft_consensus.cc:383] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:24.555956 30574 raft_consensus.cc:738] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856af570ad924de3acb21ad0f7f9a490, State: Initialized, Role: FOLLOWER
I20250624 08:02:24.556349 30574 consensus_queue.cc:260] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:24.556469 30574 raft_consensus.cc:397] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:24.556540 30574 raft_consensus.cc:491] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:24.556656 30574 raft_consensus.cc:3058] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:24.557399 30574 raft_consensus.cc:513] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:24.557703 30574 leader_election.cc:304] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 856af570ad924de3acb21ad0f7f9a490; no voters:
I20250624 08:02:24.557956 30574 leader_election.cc:290] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:24.558094 30576 raft_consensus.cc:2802] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:24.558282 30576 raft_consensus.cc:695] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 LEADER]: Becoming Leader. State: Replica: 856af570ad924de3acb21ad0f7f9a490, State: Running, Role: LEADER
I20250624 08:02:24.558552 30576 consensus_queue.cc:237] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:24.558609 30381 heartbeater.cc:499] Master 127.26.229.254:39173 was elected leader, sending a full tablet report...
I20250624 08:02:24.558764 30574 ts_tablet_manager.cc:1428] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250624 08:02:24.561157 30062 catalog_manager.cc:5582] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 reported cstate change: term changed from 0 to 1, leader changed from <none> to 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194). New cstate: current_term: 1 leader_uuid: "856af570ad924de3acb21ad0f7f9a490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:24.601707 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30031
I20250624 08:02:25.019205 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30122
I20250624 08:02:25.024678 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30253
I20250624 08:02:25.029857 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30384
I20250624 08:02:25.035421 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:39173
--webserver_interface=127.26.229.254
--webserver_port=34043
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:39173 with env {}
W20250624 08:02:25.142269 30633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:25.142500 30633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:25.142544 30633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:25.145748 30633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:25.145810 30633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:25.145833 30633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:25.145851 30633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:25.149489 30633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:39173
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=34043
--never_fsync=true
--heap_profile_path=/tmp/kudu.30633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:25.150540 30633 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:25.151665 30633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:25.158345 30638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.158407 30633 server_base.cc:1048] running on GCE node
W20250624 08:02:25.158460 30639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.158576 30641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.159101 30633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:25.159889 30633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:25.161075 30633 hybrid_clock.cc:648] HybridClock initialized: now 1750752145161056 us; error 29 us; skew 500 ppm
I20250624 08:02:25.162865 30633 webserver.cc:469] Webserver started at http://127.26.229.254:34043/ using document root <none> and password file <none>
I20250624 08:02:25.163425 30633 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:25.163482 30633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:25.166559 30633 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:25.168453 30647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:25.169426 30633 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:25.169546 30633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "997da62dfb914b9bb45a72d56c09d8f6"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:25.169895 30633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:25.182902 30633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:25.183640 30633 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:25.183825 30633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:25.190950 30633 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:39173
I20250624 08:02:25.190960 30699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:39173 every 8 connection(s)
I20250624 08:02:25.192067 30633 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:25.195721 30700 sys_catalog.cc:263] Verifying existing consensus state
I20250624 08:02:25.197177 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap starting.
I20250624 08:02:25.200857 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30633
I20250624 08:02:25.201225 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:39177
--local_ip_for_outbound_sockets=127.26.229.193
--tserver_master_addrs=127.26.229.254:39173
--webserver_port=42361
--webserver_interface=127.26.229.193
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:25.201970 30700 log.cc:826] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:25.213873 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:25.214279 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap complete.
I20250624 08:02:25.216964 30700 raft_consensus.cc:357] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.217284 30700 raft_consensus.cc:738] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997da62dfb914b9bb45a72d56c09d8f6, State: Initialized, Role: FOLLOWER
I20250624 08:02:25.217773 30700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.217892 30700 raft_consensus.cc:397] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:25.217958 30700 raft_consensus.cc:491] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:25.218072 30700 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 2 FOLLOWER]: Advancing to term 3
I20250624 08:02:25.219569 30700 raft_consensus.cc:513] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.219858 30700 leader_election.cc:304] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997da62dfb914b9bb45a72d56c09d8f6; no voters:
I20250624 08:02:25.220115 30700 leader_election.cc:290] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250624 08:02:25.220214 30704 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 08:02:25.220422 30704 raft_consensus.cc:695] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 LEADER]: Becoming Leader. State: Replica: 997da62dfb914b9bb45a72d56c09d8f6, State: Running, Role: LEADER
I20250624 08:02:25.220744 30704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.221220 30700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:25.222244 30706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997da62dfb914b9bb45a72d56c09d8f6. Latest consensus state: current_term: 3 leader_uuid: "997da62dfb914b9bb45a72d56c09d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } } }
I20250624 08:02:25.222259 30705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "997da62dfb914b9bb45a72d56c09d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } } }
I20250624 08:02:25.222370 30706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:25.222391 30705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:25.223065 30713 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:25.226766 30713 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=0f108a40c3d44c298e1291ded5ada962]
I20250624 08:02:25.227211 30713 catalog_manager.cc:671] Loaded metadata for table TestTable [id=652fcbb1183f47ac9fff8e98e9761f66]
I20250624 08:02:25.227521 30713 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=709fb731c6c84cf4ab6d1b25bdf3f2e0]
I20250624 08:02:25.228864 30713 tablet_loader.cc:96] loaded metadata for tablet 615ee19f6e234df2b717bb61609b1b44 (table TestTable2 [id=0f108a40c3d44c298e1291ded5ada962])
I20250624 08:02:25.229135 30713 tablet_loader.cc:96] loaded metadata for tablet 855d2623a4d6420a9de9a38744eed166 (table TestTable [id=652fcbb1183f47ac9fff8e98e9761f66])
I20250624 08:02:25.229367 30713 tablet_loader.cc:96] loaded metadata for tablet 9bfc26ce4f374a048595bfce459771d4 (table TestTable1 [id=709fb731c6c84cf4ab6d1b25bdf3f2e0])
I20250624 08:02:25.229502 30713 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:25.230448 30713 catalog_manager.cc:1261] Loaded cluster ID: ea9f3b2913994e4194bc2a926e42481f
I20250624 08:02:25.230556 30713 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:25.233117 30713 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:25.233963 30713 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Loaded TSK: 0
I20250624 08:02:25.235371 30713 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 08:02:25.311651 30702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:25.311872 30702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:25.311949 30702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:25.315079 30702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:25.315217 30702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:25.319010 30702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:39177
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=42361
--tserver_master_addrs=127.26.229.254:39173
--never_fsync=true
--heap_profile_path=/tmp/kudu.30702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:25.320122 30702 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:25.321216 30702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:25.328960 30726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.328971 30729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.329010 30727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.329672 30702 server_base.cc:1048] running on GCE node
I20250624 08:02:25.330020 30702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:25.330529 30702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:25.331691 30702 hybrid_clock.cc:648] HybridClock initialized: now 1750752145331665 us; error 41 us; skew 500 ppm
I20250624 08:02:25.333454 30702 webserver.cc:469] Webserver started at http://127.26.229.193:42361/ using document root <none> and password file <none>
I20250624 08:02:25.334008 30702 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:25.334074 30702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:25.337335 30702 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:25.339198 30735 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:25.340163 30702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:25.340260 30702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "50a1ce8727584f8fbd564646831ac49d"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:25.340613 30702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:25.356335 30702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:25.357007 30702 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:25.357167 30702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:25.357683 30702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:25.358839 30742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250624 08:02:25.361728 30702 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:02:25.361779 30702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250624 08:02:25.361828 30702 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:02:25.363260 30702 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:02:25.363315 30702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20250624 08:02:25.363616 30742 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap starting.
I20250624 08:02:25.375228 30702 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:39177
I20250624 08:02:25.375329 30849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:39177 every 8 connection(s)
I20250624 08:02:25.376710 30702 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:25.376888 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30702
I20250624 08:02:25.377461 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:44177
--local_ip_for_outbound_sockets=127.26.229.194
--tserver_master_addrs=127.26.229.254:39173
--webserver_port=45915
--webserver_interface=127.26.229.194
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:25.384023 30742 log.cc:826] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:25.393761 30850 heartbeater.cc:344] Connected to a master server at 127.26.229.254:39173
I20250624 08:02:25.393990 30850 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:25.394582 30850 heartbeater.cc:507] Master 127.26.229.254:39173 requested a full tablet report, sending...
I20250624 08:02:25.397260 30664 ts_manager.cc:194] Registered new tserver with Master: 50a1ce8727584f8fbd564646831ac49d (127.26.229.193:39177)
I20250624 08:02:25.399963 30664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:50861
I20250624 08:02:25.403885 30742 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:25.404217 30742 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap complete.
I20250624 08:02:25.405226 30742 ts_tablet_manager.cc:1397] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.005s
I20250624 08:02:25.407006 30742 raft_consensus.cc:357] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:25.407290 30742 raft_consensus.cc:738] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a1ce8727584f8fbd564646831ac49d, State: Initialized, Role: FOLLOWER
I20250624 08:02:25.407685 30742 consensus_queue.cc:260] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:25.407788 30742 raft_consensus.cc:397] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:25.407850 30742 raft_consensus.cc:491] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:25.407958 30742 raft_consensus.cc:3058] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:25.409603 30742 raft_consensus.cc:513] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:25.409924 30742 leader_election.cc:304] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50a1ce8727584f8fbd564646831ac49d; no voters:
I20250624 08:02:25.410202 30742 leader_election.cc:290] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 08:02:25.410332 30856 raft_consensus.cc:2802] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:25.410589 30856 raft_consensus.cc:695] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 2 LEADER]: Becoming Leader. State: Replica: 50a1ce8727584f8fbd564646831ac49d, State: Running, Role: LEADER
I20250624 08:02:25.410848 30850 heartbeater.cc:499] Master 127.26.229.254:39173 was elected leader, sending a full tablet report...
I20250624 08:02:25.411029 30742 ts_tablet_manager.cc:1428] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250624 08:02:25.411237 30856 consensus_queue.cc:237] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } }
I20250624 08:02:25.413777 30664 catalog_manager.cc:5582] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d reported cstate change: term changed from 0 to 2, leader changed from <none> to 50a1ce8727584f8fbd564646831ac49d (127.26.229.193), VOTER 50a1ce8727584f8fbd564646831ac49d (127.26.229.193) added. New cstate: current_term: 2 leader_uuid: "50a1ce8727584f8fbd564646831ac49d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } health_report { overall_health: HEALTHY } } }
W20250624 08:02:25.419523 30664 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 855d2623a4d6420a9de9a38744eed166 with cas_config_opid_index -1: no extra replica candidate found for tablet 855d2623a4d6420a9de9a38744eed166 (table TestTable [id=652fcbb1183f47ac9fff8e98e9761f66]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250624 08:02:25.487174 30852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:25.487385 30852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:25.487466 30852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:25.490495 30852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:25.490639 30852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:25.494207 30852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:44177
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=45915
--tserver_master_addrs=127.26.229.254:39173
--never_fsync=true
--heap_profile_path=/tmp/kudu.30852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:25.495368 30852 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:25.496433 30852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:25.503707 30866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.503752 30867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.503726 30869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.503966 30852 server_base.cc:1048] running on GCE node
I20250624 08:02:25.504307 30852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:25.504803 30852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:25.505965 30852 hybrid_clock.cc:648] HybridClock initialized: now 1750752145505957 us; error 24 us; skew 500 ppm
I20250624 08:02:25.507656 30852 webserver.cc:469] Webserver started at http://127.26.229.194:45915/ using document root <none> and password file <none>
I20250624 08:02:25.508191 30852 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:25.508265 30852 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:25.511359 30852 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:25.513293 30875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:25.514276 30852 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:25.514371 30852 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "856af570ad924de3acb21ad0f7f9a490"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:25.514722 30852 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:25.546254 30852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:25.546952 30852 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:25.547128 30852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:25.547674 30852 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:25.548844 30882 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250624 08:02:25.551878 30852 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:02:25.551970 30852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250624 08:02:25.552030 30852 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:02:25.553340 30852 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:02:25.553380 30852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:25.553679 30882 tablet_bootstrap.cc:492] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap starting.
I20250624 08:02:25.563966 30852 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:44177
I20250624 08:02:25.564002 30989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:44177 every 8 connection(s)
I20250624 08:02:25.564920 30852 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:25.569762 30882 log.cc:826] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:25.573961 30990 heartbeater.cc:344] Connected to a master server at 127.26.229.254:39173
I20250624 08:02:25.574266 30990 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:25.574834 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30852
I20250624 08:02:25.575028 30990 heartbeater.cc:507] Master 127.26.229.254:39173 requested a full tablet report, sending...
I20250624 08:02:25.575384 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:42011
--local_ip_for_outbound_sockets=127.26.229.195
--tserver_master_addrs=127.26.229.254:39173
--webserver_port=43379
--webserver_interface=127.26.229.195
--builtin_ntp_servers=127.26.229.212:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 08:02:25.576969 30664 ts_manager.cc:194] Registered new tserver with Master: 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194:44177)
I20250624 08:02:25.577948 30664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:56631
I20250624 08:02:25.588160 30882 tablet_bootstrap.cc:492] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:25.588538 30882 tablet_bootstrap.cc:492] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap complete.
I20250624 08:02:25.589574 30882 ts_tablet_manager.cc:1397] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.012s
I20250624 08:02:25.591564 30882 raft_consensus.cc:357] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:25.591857 30882 raft_consensus.cc:738] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856af570ad924de3acb21ad0f7f9a490, State: Initialized, Role: FOLLOWER
I20250624 08:02:25.592264 30882 consensus_queue.cc:260] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:25.592372 30882 raft_consensus.cc:397] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:25.592433 30882 raft_consensus.cc:491] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:25.592542 30882 raft_consensus.cc:3058] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:25.594162 30882 raft_consensus.cc:513] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:25.594470 30882 leader_election.cc:304] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 856af570ad924de3acb21ad0f7f9a490; no voters:
I20250624 08:02:25.594735 30882 leader_election.cc:290] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 08:02:25.594889 30995 raft_consensus.cc:2802] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:25.595081 30995 raft_consensus.cc:695] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [term 2 LEADER]: Becoming Leader. State: Replica: 856af570ad924de3acb21ad0f7f9a490, State: Running, Role: LEADER
I20250624 08:02:25.595391 30990 heartbeater.cc:499] Master 127.26.229.254:39173 was elected leader, sending a full tablet report...
I20250624 08:02:25.595425 30995 consensus_queue.cc:237] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } }
I20250624 08:02:25.595542 30882 ts_tablet_manager.cc:1428] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250624 08:02:25.597605 30664 catalog_manager.cc:5582] T 615ee19f6e234df2b717bb61609b1b44 P 856af570ad924de3acb21ad0f7f9a490 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "856af570ad924de3acb21ad0f7f9a490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 44177 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:25.635022 30804 consensus_queue.cc:237] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } }
I20250624 08:02:25.636230 30858 raft_consensus.cc:2953] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index -1 to 11, NON_VOTER 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } }
I20250624 08:02:25.638475 30650 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 855d2623a4d6420a9de9a38744eed166 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250624 08:02:25.639369 30664 catalog_manager.cc:5582] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d reported cstate change: config changed from index -1 to 11, NON_VOTER 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194) added. New cstate: current_term: 2 leader_uuid: "50a1ce8727584f8fbd564646831ac49d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250624 08:02:25.643100 30737 consensus_peers.cc:487] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d -> Peer 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194:44177): Couldn't send request to peer 856af570ad924de3acb21ad0f7f9a490. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 855d2623a4d6420a9de9a38744eed166. This is attempt 1: this message will repeat every 5th retry.
W20250624 08:02:25.686336 30994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:25.686540 30994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:25.686610 30994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:25.689666 30994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:25.689802 30994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:25.693548 30994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:42011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=43379
--tserver_master_addrs=127.26.229.254:39173
--never_fsync=true
--heap_profile_path=/tmp/kudu.30994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:25.694623 30994 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:25.695772 30994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:25.703083 31008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.703171 31010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.703249 31007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.703598 30994 server_base.cc:1048] running on GCE node
I20250624 08:02:25.703991 30994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:25.704524 30994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:25.705688 30994 hybrid_clock.cc:648] HybridClock initialized: now 1750752145705678 us; error 28 us; skew 500 ppm
I20250624 08:02:25.707500 30994 webserver.cc:469] Webserver started at http://127.26.229.195:43379/ using document root <none> and password file <none>
I20250624 08:02:25.708065 30994 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:25.708135 30994 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:25.711422 30994 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:25.713253 31016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:25.714200 30994 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:25.714294 30994 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7aadeff39afb4ba6b4e0d692791b53b2"
format_stamp: "Formatted at 2025-06-24 08:02:24 on dist-test-slave-4j8z"
I20250624 08:02:25.714655 30994 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:25.730515 30994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:25.731197 30994 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:25.731360 30994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:25.731868 30994 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:25.732961 31023 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250624 08:02:25.736027 30994 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:02:25.736094 30994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250624 08:02:25.736164 30994 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:02:25.737560 30994 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:02:25.737603 30994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:25.737912 31023 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap starting.
I20250624 08:02:25.748139 30994 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:42011
I20250624 08:02:25.748206 31130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:42011 every 8 connection(s)
I20250624 08:02:25.749152 30994 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:25.752446 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 30994
I20250624 08:02:25.753464 31023 log.cc:826] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:25.758842 31131 heartbeater.cc:344] Connected to a master server at 127.26.229.254:39173
I20250624 08:02:25.759119 31131 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:25.759685 31131 heartbeater.cc:507] Master 127.26.229.254:39173 requested a full tablet report, sending...
I20250624 08:02:25.761217 30664 ts_manager.cc:194] Registered new tserver with Master: 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195:42011)
I20250624 08:02:25.761934 30664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:60503
I20250624 08:02:25.768543 30804 consensus_queue.cc:237] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } }
I20250624 08:02:25.769407 30858 raft_consensus.cc:2953] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } } }
W20250624 08:02:25.770104 30737 consensus_peers.cc:487] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d -> Peer 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194:44177): Couldn't send request to peer 856af570ad924de3acb21ad0f7f9a490. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 855d2623a4d6420a9de9a38744eed166. This is attempt 1: this message will repeat every 5th retry.
I20250624 08:02:25.771027 30650 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 855d2623a4d6420a9de9a38744eed166 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20250624 08:02:25.771797 30664 catalog_manager.cc:5582] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d reported cstate change: config changed from index 11 to 12, NON_VOTER 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195) added. New cstate: current_term: 2 leader_uuid: "50a1ce8727584f8fbd564646831ac49d" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250624 08:02:25.772202 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:25.773543 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:25.775365 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 50a1ce8727584f8fbd564646831ac49d to finish bootstrapping
W20250624 08:02:25.776273 30739 consensus_peers.cc:487] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d -> Peer 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195:42011): Couldn't send request to peer 7aadeff39afb4ba6b4e0d692791b53b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 855d2623a4d6420a9de9a38744eed166. This is attempt 1: this message will repeat every 5th retry.
I20250624 08:02:25.777276 31023 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:25.777678 31023 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap complete.
I20250624 08:02:25.778916 31023 ts_tablet_manager.cc:1397] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.007s
I20250624 08:02:25.780951 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 856af570ad924de3acb21ad0f7f9a490 to finish bootstrapping
I20250624 08:02:25.781345 31023 raft_consensus.cc:357] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:25.781744 31023 raft_consensus.cc:738] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aadeff39afb4ba6b4e0d692791b53b2, State: Initialized, Role: FOLLOWER
I20250624 08:02:25.782219 31023 consensus_queue.cc:260] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:25.782342 31023 raft_consensus.cc:397] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:25.782410 31023 raft_consensus.cc:491] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:25.782518 31023 raft_consensus.cc:3058] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:25.784359 31023 raft_consensus.cc:513] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:25.784664 31023 leader_election.cc:304] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aadeff39afb4ba6b4e0d692791b53b2; no voters:
I20250624 08:02:25.784988 31023 leader_election.cc:290] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 08:02:25.785107 31138 raft_consensus.cc:2802] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:25.785331 31138 raft_consensus.cc:695] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 LEADER]: Becoming Leader. State: Replica: 7aadeff39afb4ba6b4e0d692791b53b2, State: Running, Role: LEADER
I20250624 08:02:25.785671 31131 heartbeater.cc:499] Master 127.26.229.254:39173 was elected leader, sending a full tablet report...
I20250624 08:02:25.785651 31138 consensus_queue.cc:237] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } }
I20250624 08:02:25.785789 31023 ts_tablet_manager.cc:1428] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250624 08:02:25.786156 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7aadeff39afb4ba6b4e0d692791b53b2 to finish bootstrapping
I20250624 08:02:25.787696 30664 catalog_manager.cc:5582] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 reported cstate change: term changed from 0 to 2, leader changed from <none> to 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195), VOTER 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195) added. New cstate: current_term: 2 leader_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:25.794682 31085 consensus_queue.cc:237] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } }
I20250624 08:02:25.795907 31140 raft_consensus.cc:2953] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index -1 to 14, NON_VOTER 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194) added. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } }
I20250624 08:02:25.798244 30651 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9bfc26ce4f374a048595bfce459771d4 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250624 08:02:25.798606 30664 catalog_manager.cc:5582] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 reported cstate change: config changed from index -1 to 14, NON_VOTER 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194) added. New cstate: current_term: 2 leader_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250624 08:02:25.801635 31085 consensus_queue.cc:237] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 12, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } }
I20250624 08:02:25.802456 31139 raft_consensus.cc:2953] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 50a1ce8727584f8fbd564646831ac49d (127.26.229.193) added. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } } }
W20250624 08:02:25.803187 31018 consensus_peers.cc:487] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 -> Peer 856af570ad924de3acb21ad0f7f9a490 (127.26.229.194:44177): Couldn't send request to peer 856af570ad924de3acb21ad0f7f9a490. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bfc26ce4f374a048595bfce459771d4. This is attempt 1: this message will repeat every 5th retry.
I20250624 08:02:25.804261 30651 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9bfc26ce4f374a048595bfce459771d4 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250624 08:02:25.806530 31019 consensus_peers.cc:487] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 -> Peer 50a1ce8727584f8fbd564646831ac49d (127.26.229.193:39177): Couldn't send request to peer 50a1ce8727584f8fbd564646831ac49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bfc26ce4f374a048595bfce459771d4. This is attempt 1: this message will repeat every 5th retry.
I20250624 08:02:25.807752 30664 catalog_manager.cc:5582] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2 reported cstate change: config changed from index 14 to 15, NON_VOTER 50a1ce8727584f8fbd564646831ac49d (127.26.229.193) added. New cstate: current_term: 2 leader_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250624 08:02:25.846320 30784 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:25.846637 31065 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:25.848130 30924 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
997da62dfb914b9bb45a72d56c09d8f6 | 127.26.229.254:39173 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:42031 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
50a1ce8727584f8fbd564646831ac49d | 127.26.229.193:39177 | HEALTHY | <none> | 1 | 0
7aadeff39afb4ba6b4e0d692791b53b2 | 127.26.229.195:42011 | HEALTHY | <none> | 1 | 0
856af570ad924de3acb21ad0f7f9a490 | 127.26.229.194:44177 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.26.229.193 | experimental | 127.26.229.193:39177
local_ip_for_outbound_sockets | 127.26.229.194 | experimental | 127.26.229.194:44177
local_ip_for_outbound_sockets | 127.26.229.195 | experimental | 127.26.229.195:42011
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.26.229.193:39177
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.26.229.194:44177
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.26.229.195:42011
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:42031 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 08:02:25.873854 27543 log_verifier.cc:126] Checking tablet 615ee19f6e234df2b717bb61609b1b44
I20250624 08:02:25.884609 27543 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 615ee19f6e234df2b717bb61609b1b44
I20250624 08:02:25.884701 27543 log_verifier.cc:126] Checking tablet 855d2623a4d6420a9de9a38744eed166
I20250624 08:02:25.892840 27543 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 855d2623a4d6420a9de9a38744eed166
I20250624 08:02:25.892907 27543 log_verifier.cc:126] Checking tablet 9bfc26ce4f374a048595bfce459771d4
I20250624 08:02:25.901432 27543 log_verifier.cc:177] Verified matching terms for 15 ops in tablet 9bfc26ce4f374a048595bfce459771d4
I20250624 08:02:25.901587 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30633
I20250624 08:02:25.908411 27543 minidump.cc:252] Setting minidump size limit to 20M
I20250624 08:02:25.908910 27543 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:25.909368 27543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:25.914033 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:25.914104 31171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.914162 27543 server_base.cc:1048] running on GCE node
W20250624 08:02:25.914263 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:25.914791 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250624 08:02:25.914955 27543 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250624 08:02:25.915000 27543 hybrid_clock.cc:648] HybridClock initialized: now 1750752145914998 us; error 0 us; skew 500 ppm
I20250624 08:02:25.916347 27543 webserver.cc:469] Webserver started at http://0.0.0.0:45465/ using document root <none> and password file <none>
I20250624 08:02:25.916947 27543 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:25.916999 27543 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:25.919669 27543 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250624 08:02:25.921651 31176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:25.922561 27543 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250624 08:02:25.922686 27543 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "997da62dfb914b9bb45a72d56c09d8f6"
format_stamp: "Formatted at 2025-06-24 08:02:23 on dist-test-slave-4j8z"
I20250624 08:02:25.922978 27543 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:25.932155 27543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:25.932708 27543 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:25.932858 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:25.935536 27543 sys_catalog.cc:263] Verifying existing consensus state
W20250624 08:02:25.936232 27543 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.26.229.254:39173 exists but no master address supplied!
I20250624 08:02:25.937466 27543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap starting.
I20250624 08:02:25.949764 27543 log.cc:826] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:25.964810 27543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=13 ignored=0} mutations{seen=17 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:25.965234 27543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6: Bootstrap complete.
I20250624 08:02:25.968343 27543 raft_consensus.cc:357] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.968513 27543 raft_consensus.cc:738] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997da62dfb914b9bb45a72d56c09d8f6, State: Initialized, Role: FOLLOWER
I20250624 08:02:25.968959 27543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.969084 27543 raft_consensus.cc:397] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:25.969154 27543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:25.969281 27543 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 3 FOLLOWER]: Advancing to term 4
I20250624 08:02:25.970631 27543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.970964 27543 leader_election.cc:304] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997da62dfb914b9bb45a72d56c09d8f6; no voters:
I20250624 08:02:25.971236 27543 leader_election.cc:290] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250624 08:02:25.971365 31183 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 4 FOLLOWER]: Leader election won for term 4
I20250624 08:02:25.971606 31183 raft_consensus.cc:695] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 4 LEADER]: Becoming Leader. State: Replica: 997da62dfb914b9bb45a72d56c09d8f6, State: Running, Role: LEADER
I20250624 08:02:25.971923 31183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } }
I20250624 08:02:25.973173 31184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "997da62dfb914b9bb45a72d56c09d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } } }
I20250624 08:02:25.973289 31184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:25.973238 31185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997da62dfb914b9bb45a72d56c09d8f6. Latest consensus state: current_term: 4 leader_uuid: "997da62dfb914b9bb45a72d56c09d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997da62dfb914b9bb45a72d56c09d8f6" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 39173 } } }
I20250624 08:02:25.973425 31185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:25.980176 27543 tablet_replica.cc:331] stopping tablet replica
I20250624 08:02:25.980424 27543 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 4 LEADER]: Raft consensus shutting down.
I20250624 08:02:25.980562 27543 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 997da62dfb914b9bb45a72d56c09d8f6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250624 08:02:25.981779 27543 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250624 08:02:25.982077 27543 acceptor_pool.cc:194] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250624 08:02:25.993520 27543 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250624 08:02:26.279387 31225 ts_tablet_manager.cc:927] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: Initiating tablet copy from peer 50a1ce8727584f8fbd564646831ac49d (127.26.229.193:39177)
I20250624 08:02:26.279937 31225 tablet_copy_client.cc:323] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Beginning tablet copy session from remote peer at address 127.26.229.193:39177
I20250624 08:02:26.283874 30824 tablet_copy_service.cc:140] P 50a1ce8727584f8fbd564646831ac49d: Received BeginTabletCopySession request for tablet 855d2623a4d6420a9de9a38744eed166 from peer 856af570ad924de3acb21ad0f7f9a490 ({username='slave'} at 127.26.229.194:43619)
I20250624 08:02:26.284034 30824 tablet_copy_service.cc:161] P 50a1ce8727584f8fbd564646831ac49d: Beginning new tablet copy session on tablet 855d2623a4d6420a9de9a38744eed166 from peer 856af570ad924de3acb21ad0f7f9a490 at {username='slave'} at 127.26.229.194:43619: session id = 856af570ad924de3acb21ad0f7f9a490-855d2623a4d6420a9de9a38744eed166
I20250624 08:02:26.285393 30824 tablet_copy_source_session.cc:215] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 08:02:26.288300 31225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855d2623a4d6420a9de9a38744eed166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:26.290977 31225 tablet_copy_client.cc:806] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Starting download of 0 data blocks...
I20250624 08:02:26.291150 31225 tablet_copy_client.cc:670] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Starting download of 1 WAL segments...
I20250624 08:02:26.292744 31225 tablet_copy_client.cc:538] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 08:02:26.293789 31225 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap starting.
I20250624 08:02:26.308485 31225 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:26.308749 31225 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap complete.
I20250624 08:02:26.308910 31225 ts_tablet_manager.cc:1397] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20250624 08:02:26.309392 31225 raft_consensus.cc:357] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } }
I20250624 08:02:26.309489 31225 raft_consensus.cc:738] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 856af570ad924de3acb21ad0f7f9a490, State: Initialized, Role: LEARNER
I20250624 08:02:26.309589 31225 consensus_queue.cc:260] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } }
I20250624 08:02:26.309937 31225 ts_tablet_manager.cc:1428] T 855d2623a4d6420a9de9a38744eed166 P 856af570ad924de3acb21ad0f7f9a490: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250624 08:02:26.310634 30824 tablet_copy_service.cc:342] P 50a1ce8727584f8fbd564646831ac49d: Request end of tablet copy session 856af570ad924de3acb21ad0f7f9a490-855d2623a4d6420a9de9a38744eed166 received from {username='slave'} at 127.26.229.194:43619
I20250624 08:02:26.310766 30824 tablet_copy_service.cc:434] P 50a1ce8727584f8fbd564646831ac49d: ending tablet copy session 856af570ad924de3acb21ad0f7f9a490-855d2623a4d6420a9de9a38744eed166 on tablet 855d2623a4d6420a9de9a38744eed166 with peer 856af570ad924de3acb21ad0f7f9a490
W20250624 08:02:26.310884 30990 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:39173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:39173: connect: Connection refused (error 111)
I20250624 08:02:26.378628 31232 ts_tablet_manager.cc:927] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: Initiating tablet copy from peer 50a1ce8727584f8fbd564646831ac49d (127.26.229.193:39177)
I20250624 08:02:26.379204 31232 tablet_copy_client.cc:323] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: tablet copy: Beginning tablet copy session from remote peer at address 127.26.229.193:39177
I20250624 08:02:26.379873 30824 tablet_copy_service.cc:140] P 50a1ce8727584f8fbd564646831ac49d: Received BeginTabletCopySession request for tablet 855d2623a4d6420a9de9a38744eed166 from peer 7aadeff39afb4ba6b4e0d692791b53b2 ({username='slave'} at 127.26.229.195:35767)
I20250624 08:02:26.379974 30824 tablet_copy_service.cc:161] P 50a1ce8727584f8fbd564646831ac49d: Beginning new tablet copy session on tablet 855d2623a4d6420a9de9a38744eed166 from peer 7aadeff39afb4ba6b4e0d692791b53b2 at {username='slave'} at 127.26.229.195:35767: session id = 7aadeff39afb4ba6b4e0d692791b53b2-855d2623a4d6420a9de9a38744eed166
I20250624 08:02:26.380776 30824 tablet_copy_source_session.cc:215] T 855d2623a4d6420a9de9a38744eed166 P 50a1ce8727584f8fbd564646831ac49d: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 08:02:26.382244 31232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855d2623a4d6420a9de9a38744eed166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:26.385718 31232 tablet_copy_client.cc:806] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: tablet copy: Starting download of 0 data blocks...
I20250624 08:02:26.385910 31232 tablet_copy_client.cc:670] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: tablet copy: Starting download of 1 WAL segments...
I20250624 08:02:26.387569 31232 tablet_copy_client.cc:538] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 08:02:26.388994 31232 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap starting.
I20250624 08:02:26.400696 31225 ts_tablet_manager.cc:927] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: Initiating tablet copy from peer 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195:42011)
I20250624 08:02:26.401002 31225 tablet_copy_client.cc:323] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Beginning tablet copy session from remote peer at address 127.26.229.195:42011
I20250624 08:02:26.405143 31105 tablet_copy_service.cc:140] P 7aadeff39afb4ba6b4e0d692791b53b2: Received BeginTabletCopySession request for tablet 9bfc26ce4f374a048595bfce459771d4 from peer 856af570ad924de3acb21ad0f7f9a490 ({username='slave'} at 127.26.229.194:47913)
I20250624 08:02:26.405283 31105 tablet_copy_service.cc:161] P 7aadeff39afb4ba6b4e0d692791b53b2: Beginning new tablet copy session on tablet 9bfc26ce4f374a048595bfce459771d4 from peer 856af570ad924de3acb21ad0f7f9a490 at {username='slave'} at 127.26.229.194:47913: session id = 856af570ad924de3acb21ad0f7f9a490-9bfc26ce4f374a048595bfce459771d4
I20250624 08:02:26.406411 31105 tablet_copy_source_session.cc:215] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 08:02:26.407451 31225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfc26ce4f374a048595bfce459771d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:26.408358 31232 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:26.408572 31232 tablet_bootstrap.cc:492] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: Bootstrap complete.
I20250624 08:02:26.408711 31232 ts_tablet_manager.cc:1397] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20250624 08:02:26.409205 31232 raft_consensus.cc:357] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } }
I20250624 08:02:26.409312 31232 raft_consensus.cc:738] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7aadeff39afb4ba6b4e0d692791b53b2, State: Initialized, Role: LEARNER
I20250624 08:02:26.409397 31232 consensus_queue.cc:260] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 39177 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: NON_VOTER last_known_addr { host: "127.26.229.195" port: 42011 } attrs { promote: true } }
I20250624 08:02:26.409466 31225 tablet_copy_client.cc:806] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Starting download of 0 data blocks...
I20250624 08:02:26.409574 31225 tablet_copy_client.cc:670] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Starting download of 1 WAL segments...
I20250624 08:02:26.409647 31232 ts_tablet_manager.cc:1428] T 855d2623a4d6420a9de9a38744eed166 P 7aadeff39afb4ba6b4e0d692791b53b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:26.410866 30824 tablet_copy_service.cc:342] P 50a1ce8727584f8fbd564646831ac49d: Request end of tablet copy session 7aadeff39afb4ba6b4e0d692791b53b2-855d2623a4d6420a9de9a38744eed166 received from {username='slave'} at 127.26.229.195:35767
I20250624 08:02:26.410971 30824 tablet_copy_service.cc:434] P 50a1ce8727584f8fbd564646831ac49d: ending tablet copy session 7aadeff39afb4ba6b4e0d692791b53b2-855d2623a4d6420a9de9a38744eed166 on tablet 855d2623a4d6420a9de9a38744eed166 with peer 7aadeff39afb4ba6b4e0d692791b53b2
I20250624 08:02:26.411192 31225 tablet_copy_client.cc:538] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250624 08:02:26.411578 31131 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:39173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:39173: connect: Connection refused (error 111)
I20250624 08:02:26.412392 31225 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap starting.
I20250624 08:02:26.412779 31245 ts_tablet_manager.cc:927] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: Initiating tablet copy from peer 7aadeff39afb4ba6b4e0d692791b53b2 (127.26.229.195:42011)
I20250624 08:02:26.413208 31245 tablet_copy_client.cc:323] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: tablet copy: Beginning tablet copy session from remote peer at address 127.26.229.195:42011
I20250624 08:02:26.413650 31105 tablet_copy_service.cc:140] P 7aadeff39afb4ba6b4e0d692791b53b2: Received BeginTabletCopySession request for tablet 9bfc26ce4f374a048595bfce459771d4 from peer 50a1ce8727584f8fbd564646831ac49d ({username='slave'} at 127.26.229.193:40771)
I20250624 08:02:26.413708 31105 tablet_copy_service.cc:161] P 7aadeff39afb4ba6b4e0d692791b53b2: Beginning new tablet copy session on tablet 9bfc26ce4f374a048595bfce459771d4 from peer 50a1ce8727584f8fbd564646831ac49d at {username='slave'} at 127.26.229.193:40771: session id = 50a1ce8727584f8fbd564646831ac49d-9bfc26ce4f374a048595bfce459771d4
I20250624 08:02:26.414189 31105 tablet_copy_source_session.cc:215] T 9bfc26ce4f374a048595bfce459771d4 P 7aadeff39afb4ba6b4e0d692791b53b2: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 08:02:26.415237 31245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfc26ce4f374a048595bfce459771d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:26.417323 31245 tablet_copy_client.cc:806] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: tablet copy: Starting download of 0 data blocks...
I20250624 08:02:26.417471 31245 tablet_copy_client.cc:670] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: tablet copy: Starting download of 1 WAL segments...
I20250624 08:02:26.418552 31245 tablet_copy_client.cc:538] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 08:02:26.419697 31245 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap starting.
I20250624 08:02:26.436518 31245 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:26.436758 31245 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: Bootstrap complete.
I20250624 08:02:26.436906 31245 ts_tablet_manager.cc:1397] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250624 08:02:26.437402 31245 raft_consensus.cc:357] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } }
I20250624 08:02:26.437508 31245 raft_consensus.cc:738] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 50a1ce8727584f8fbd564646831ac49d, State: Initialized, Role: LEARNER
I20250624 08:02:26.437582 31225 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:26.437630 31245 consensus_queue.cc:260] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } }
I20250624 08:02:26.437834 31225 tablet_bootstrap.cc:492] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: Bootstrap complete.
I20250624 08:02:26.437887 31245 ts_tablet_manager.cc:1428] T 9bfc26ce4f374a048595bfce459771d4 P 50a1ce8727584f8fbd564646831ac49d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:26.437973 31225 ts_tablet_manager.cc:1397] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.000s
I20250624 08:02:26.438406 31225 raft_consensus.cc:357] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } }
I20250624 08:02:26.438503 31225 raft_consensus.cc:738] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 856af570ad924de3acb21ad0f7f9a490, State: Initialized, Role: LEARNER
I20250624 08:02:26.438582 31225 consensus_queue.cc:260] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "7aadeff39afb4ba6b4e0d692791b53b2" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 42011 } } peers { permanent_uuid: "856af570ad924de3acb21ad0f7f9a490" member_type: NON_VOTER last_known_addr { host: "127.26.229.194" port: 44177 } attrs { promote: true } } peers { permanent_uuid: "50a1ce8727584f8fbd564646831ac49d" member_type: NON_VOTER last_known_addr { host: "127.26.229.193" port: 39177 } attrs { promote: true } }
I20250624 08:02:26.438879 31225 ts_tablet_manager.cc:1428] T 9bfc26ce4f374a048595bfce459771d4 P 856af570ad924de3acb21ad0f7f9a490: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250624 08:02:26.438874 31105 tablet_copy_service.cc:342] P 7aadeff39afb4ba6b4e0d692791b53b2: Request end of tablet copy session 50a1ce8727584f8fbd564646831ac49d-9bfc26ce4f374a048595bfce459771d4 received from {username='slave'} at 127.26.229.193:40771
I20250624 08:02:26.438971 31105 tablet_copy_service.cc:434] P 7aadeff39afb4ba6b4e0d692791b53b2: ending tablet copy session 50a1ce8727584f8fbd564646831ac49d-9bfc26ce4f374a048595bfce459771d4 on tablet 9bfc26ce4f374a048595bfce459771d4 with peer 50a1ce8727584f8fbd564646831ac49d
W20250624 08:02:26.439662 30850 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:39173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:39173: connect: Connection refused (error 111)
I20250624 08:02:26.439934 31105 tablet_copy_service.cc:342] P 7aadeff39afb4ba6b4e0d692791b53b2: Request end of tablet copy session 856af570ad924de3acb21ad0f7f9a490-9bfc26ce4f374a048595bfce459771d4 received from {username='slave'} at 127.26.229.194:47913
I20250624 08:02:26.440011 31105 tablet_copy_service.cc:434] P 7aadeff39afb4ba6b4e0d692791b53b2: ending tablet copy session 856af570ad924de3acb21ad0f7f9a490-9bfc26ce4f374a048595bfce459771d4 on tablet 9bfc26ce4f374a048595bfce459771d4 with peer 856af570ad924de3acb21ad0f7f9a490
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/kudu-admin-test.cc:3908: Failure
Value of: stdout2
Expected: has substring "Rebuilt from 3 replicas, of which 0 had errors"
Actual: "Rebuild Report\nTablet Servers\n address | status\n----------------------+--------\n 127.26.229.193:39177 | OK\n 127.26.229.194:44177 | OK\n 127.26.229.195:42011 | OK\nRebuilt from 3 tablet servers, of which 0 had errors\n\nReplicas\n table | tablet | tablet server | status\n-----------+----------------------------------+----------------------+--------\n TestTable | 855d2623a4d6420a9de9a38744eed166 | 127.26.229.193:39177 | OK\nRebuilt from 1 replicas, of which 0 had errors\n"
I20250624 08:02:26.558475 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30702
I20250624 08:02:26.564972 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30852
I20250624 08:02:26.570469 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 30994
2025-06-24T08:02:26Z chronyd exiting
I20250624 08:02:26.587077 27543 test_util.cc:183] -----------------------------------------------
I20250624 08:02:26.587137 27543 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750752133635324-27543-0
[ FAILED ] AdminCliTest.TestRebuildTables (2977 ms)
[----------] 5 tests from AdminCliTest (12942 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250624 08:02:26.587725 27543 test_util.cc:276] Using random seed: -1077313462
I20250624 08:02:26.588466 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:26.588505 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:26.588517 27543 ts_itest-base.cc:117] 5 tablet servers
I20250624 08:02:26.588528 27543 ts_itest-base.cc:118] 3 replicas per TS
I20250624 08:02:26.588539 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:26Z Disabled control of system clock
I20250624 08:02:26.596819 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:45987
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:45987
--raft_prepare_replacement_before_eviction=true with env {}
W20250624 08:02:26.695586 31257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:26.695806 31257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:26.695876 31257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:26.698879 31257 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 08:02:26.698943 31257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:26.698979 31257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:26.699021 31257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:26.699056 31257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:26.702662 31257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:45987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:45987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:26.703749 31257 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:26.704804 31257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:26.710776 31265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:26.710772 31262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:26.710769 31263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:26.711059 31257 server_base.cc:1048] running on GCE node
I20250624 08:02:26.711513 31257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:26.712303 31257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:26.713459 31257 hybrid_clock.cc:648] HybridClock initialized: now 1750752146713444 us; error 35 us; skew 500 ppm
I20250624 08:02:26.715262 31257 webserver.cc:469] Webserver started at http://127.26.229.254:38091/ using document root <none> and password file <none>
I20250624 08:02:26.715770 31257 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:26.715873 31257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:26.716090 31257 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:26.717633 31257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1"
format_stamp: "Formatted at 2025-06-24 08:02:26 on dist-test-slave-4j8z"
I20250624 08:02:26.718122 31257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1"
format_stamp: "Formatted at 2025-06-24 08:02:26 on dist-test-slave-4j8z"
I20250624 08:02:26.721928 31257 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:26.724014 31271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:26.724951 31257 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:26.725051 31257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1"
format_stamp: "Formatted at 2025-06-24 08:02:26 on dist-test-slave-4j8z"
I20250624 08:02:26.725129 31257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:26.751504 31257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:26.752120 31257 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:26.752295 31257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:26.759136 31257 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:45987
I20250624 08:02:26.759151 31323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:45987 every 8 connection(s)
I20250624 08:02:26.760099 31257 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:26.761879 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 31257
I20250624 08:02:26.761989 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:26.762626 31324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:26.768440 31324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1: Bootstrap starting.
I20250624 08:02:26.770413 31324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:26.771121 31324 log.cc:826] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:26.772625 31324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1: No bootstrap required, opened a new log
I20250624 08:02:26.774854 31324 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 45987 } }
I20250624 08:02:26.775038 31324 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:26.775087 31324 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f40b392b9ca4a4c82e58a0f9a42cfd1, State: Initialized, Role: FOLLOWER
I20250624 08:02:26.775483 31324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 45987 } }
I20250624 08:02:26.775590 31324 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:26.775653 31324 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:26.775758 31324 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:26.776463 31324 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 45987 } }
I20250624 08:02:26.776772 31324 leader_election.cc:304] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f40b392b9ca4a4c82e58a0f9a42cfd1; no voters:
I20250624 08:02:26.777014 31324 leader_election.cc:290] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:26.777166 31329 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:26.777444 31329 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [term 1 LEADER]: Becoming Leader. State: Replica: 9f40b392b9ca4a4c82e58a0f9a42cfd1, State: Running, Role: LEADER
I20250624 08:02:26.777719 31329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 45987 } }
I20250624 08:02:26.778059 31324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:26.779356 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f40b392b9ca4a4c82e58a0f9a42cfd1. Latest consensus state: current_term: 1 leader_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 45987 } } }
I20250624 08:02:26.779378 31330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f40b392b9ca4a4c82e58a0f9a42cfd1" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 45987 } } }
I20250624 08:02:26.779470 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:26.779475 31330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:26.779819 31338 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:26.782318 31338 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:26.786801 31338 catalog_manager.cc:1349] Generated new cluster ID: 815027980fc64dbd9e0485fdd4af8a00
I20250624 08:02:26.786870 31338 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:26.816756 31338 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:26.817595 31338 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:26.826020 31338 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9f40b392b9ca4a4c82e58a0f9a42cfd1: Generated new TSK 0
I20250624 08:02:26.826679 31338 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:26.836902 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--builtin_ntp_servers=127.26.229.212:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250624 08:02:26.938889 31348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:26.939090 31348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:26.939133 31348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:26.942196 31348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 08:02:26.942252 31348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:26.942350 31348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:26.945725 31348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--never_fsync=true
--heap_profile_path=/tmp/kudu.31348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:26.946681 31348 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:26.947729 31348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:26.955173 31353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:26.955166 31354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:26.955353 31348 server_base.cc:1048] running on GCE node
W20250624 08:02:26.955166 31356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:26.955888 31348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:26.956451 31348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:26.957631 31348 hybrid_clock.cc:648] HybridClock initialized: now 1750752146957619 us; error 26 us; skew 500 ppm
I20250624 08:02:26.959578 31348 webserver.cc:469] Webserver started at http://127.26.229.193:36833/ using document root <none> and password file <none>
I20250624 08:02:26.960150 31348 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:26.960206 31348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:26.960395 31348 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:26.961949 31348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5bdebf9256864f06b8ed90ecc00e1c54"
format_stamp: "Formatted at 2025-06-24 08:02:26 on dist-test-slave-4j8z"
I20250624 08:02:26.962396 31348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5bdebf9256864f06b8ed90ecc00e1c54"
format_stamp: "Formatted at 2025-06-24 08:02:26 on dist-test-slave-4j8z"
I20250624 08:02:26.965667 31348 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:26.967800 31362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:26.968794 31348 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:26.968909 31348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5bdebf9256864f06b8ed90ecc00e1c54"
format_stamp: "Formatted at 2025-06-24 08:02:26 on dist-test-slave-4j8z"
I20250624 08:02:26.969033 31348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:26.994462 31348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:26.995111 31348 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:26.995296 31348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:26.995839 31348 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:26.996735 31348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:26.996783 31348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:26.996835 31348 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:26.996876 31348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.005553 31348 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:34347
I20250624 08:02:27.005638 31475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:34347 every 8 connection(s)
I20250624 08:02:27.006492 31348 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:27.012755 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 31348
I20250624 08:02:27.012912 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:27.015426 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--builtin_ntp_servers=127.26.229.212:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 08:02:27.016269 31476 heartbeater.cc:344] Connected to a master server at 127.26.229.254:45987
I20250624 08:02:27.016521 31476 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:27.017165 31476 heartbeater.cc:507] Master 127.26.229.254:45987 requested a full tablet report, sending...
I20250624 08:02:27.018498 31288 ts_manager.cc:194] Registered new tserver with Master: 5bdebf9256864f06b8ed90ecc00e1c54 (127.26.229.193:34347)
I20250624 08:02:27.019624 31288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:40945
W20250624 08:02:27.116309 31479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:27.116566 31479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:27.116644 31479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:27.120648 31479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 08:02:27.120761 31479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:27.120910 31479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:27.125176 31479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--never_fsync=true
--heap_profile_path=/tmp/kudu.31479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:27.126139 31479 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:27.127213 31479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:27.134258 31485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:27.134276 31484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:27.134330 31479 server_base.cc:1048] running on GCE node
W20250624 08:02:27.134276 31487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:27.134752 31479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:27.135269 31479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:27.136422 31479 hybrid_clock.cc:648] HybridClock initialized: now 1750752147136410 us; error 29 us; skew 500 ppm
I20250624 08:02:27.138113 31479 webserver.cc:469] Webserver started at http://127.26.229.194:45431/ using document root <none> and password file <none>
I20250624 08:02:27.138653 31479 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:27.138711 31479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:27.138923 31479 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:27.140408 31479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "db284502a3b14fffb1ed9472317d632a"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.140880 31479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "db284502a3b14fffb1ed9472317d632a"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.143990 31479 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 08:02:27.145908 31493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.146848 31479 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:27.146945 31479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "db284502a3b14fffb1ed9472317d632a"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.147027 31479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:27.168841 31479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:27.169442 31479 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:27.169612 31479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:27.170142 31479 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:27.171096 31479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:27.171155 31479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.171216 31479 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:27.171262 31479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.179975 31479 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:42501
I20250624 08:02:27.180011 31606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:42501 every 8 connection(s)
I20250624 08:02:27.180879 31479 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 08:02:27.181608 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 31479
I20250624 08:02:27.181722 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 08:02:27.184247 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--builtin_ntp_servers=127.26.229.212:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 08:02:27.190470 31607 heartbeater.cc:344] Connected to a master server at 127.26.229.254:45987
I20250624 08:02:27.190683 31607 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:27.191227 31607 heartbeater.cc:507] Master 127.26.229.254:45987 requested a full tablet report, sending...
I20250624 08:02:27.192230 31288 ts_manager.cc:194] Registered new tserver with Master: db284502a3b14fffb1ed9472317d632a (127.26.229.194:42501)
I20250624 08:02:27.192780 31288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:38509
W20250624 08:02:27.285337 31610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:27.285511 31610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:27.285573 31610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:27.289016 31610 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 08:02:27.289104 31610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:27.289192 31610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:27.292794 31610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--never_fsync=true
--heap_profile_path=/tmp/kudu.31610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:27.293738 31610 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:27.294696 31610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:27.301611 31615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:27.301642 31616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:27.301631 31618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:27.301844 31610 server_base.cc:1048] running on GCE node
I20250624 08:02:27.302213 31610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:27.302693 31610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:27.303845 31610 hybrid_clock.cc:648] HybridClock initialized: now 1750752147303826 us; error 39 us; skew 500 ppm
I20250624 08:02:27.305577 31610 webserver.cc:469] Webserver started at http://127.26.229.195:34533/ using document root <none> and password file <none>
I20250624 08:02:27.306102 31610 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:27.306154 31610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:27.306334 31610 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:27.307826 31610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8b493f87f687454cb8c291ed2b01bcd8"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.308274 31610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8b493f87f687454cb8c291ed2b01bcd8"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.311508 31610 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250624 08:02:27.313535 31624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.314420 31610 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250624 08:02:27.314530 31610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8b493f87f687454cb8c291ed2b01bcd8"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.314606 31610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:27.329735 31610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:27.330297 31610 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:27.330461 31610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:27.331049 31610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:27.331984 31610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:27.332039 31610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.332108 31610 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:27.332140 31610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.341315 31610 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:37955
I20250624 08:02:27.341342 31737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:37955 every 8 connection(s)
I20250624 08:02:27.342244 31610 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 08:02:27.350507 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 31610
I20250624 08:02:27.350644 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 08:02:27.350828 31738 heartbeater.cc:344] Connected to a master server at 127.26.229.254:45987
I20250624 08:02:27.351079 31738 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:27.351639 31738 heartbeater.cc:507] Master 127.26.229.254:45987 requested a full tablet report, sending...
I20250624 08:02:27.352684 31288 ts_manager.cc:194] Registered new tserver with Master: 8b493f87f687454cb8c291ed2b01bcd8 (127.26.229.195:37955)
I20250624 08:02:27.353003 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.196:0
--local_ip_for_outbound_sockets=127.26.229.196
--webserver_interface=127.26.229.196
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--builtin_ntp_servers=127.26.229.212:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 08:02:27.353413 31288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:38091
W20250624 08:02:27.455359 31741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:27.455592 31741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:27.455638 31741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:27.458941 31741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 08:02:27.459024 31741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:27.459098 31741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.196
I20250624 08:02:27.462921 31741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.26.229.196
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--never_fsync=true
--heap_profile_path=/tmp/kudu.31741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.196
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:27.463932 31741 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:27.464987 31741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:27.472493 31747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:27.472559 31749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:27.472673 31741 server_base.cc:1048] running on GCE node
W20250624 08:02:27.472477 31746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:27.473122 31741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:27.473651 31741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:27.474839 31741 hybrid_clock.cc:648] HybridClock initialized: now 1750752147474831 us; error 25 us; skew 500 ppm
I20250624 08:02:27.476568 31741 webserver.cc:469] Webserver started at http://127.26.229.196:32855/ using document root <none> and password file <none>
I20250624 08:02:27.477116 31741 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:27.477178 31741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:27.477352 31741 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:27.478973 31741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3ac54d41522441fd83d4e11e6b2b20e4"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.479478 31741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3ac54d41522441fd83d4e11e6b2b20e4"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.482707 31741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250624 08:02:27.484743 31755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.485745 31741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:27.485832 31741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3ac54d41522441fd83d4e11e6b2b20e4"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.485965 31741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:27.498584 31741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:27.499233 31741 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:27.499399 31741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:27.499907 31741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:27.500766 31741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:27.500811 31741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.500849 31741 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:27.500892 31741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.509585 31741 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.196:32795
I20250624 08:02:27.509616 31868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.196:32795 every 8 connection(s)
I20250624 08:02:27.510538 31741 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250624 08:02:27.519130 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 31741
I20250624 08:02:27.519243 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250624 08:02:27.519723 31869 heartbeater.cc:344] Connected to a master server at 127.26.229.254:45987
I20250624 08:02:27.519959 31869 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:27.520484 31869 heartbeater.cc:507] Master 127.26.229.254:45987 requested a full tablet report, sending...
I20250624 08:02:27.521422 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.197:0
--local_ip_for_outbound_sockets=127.26.229.197
--webserver_interface=127.26.229.197
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--builtin_ntp_servers=127.26.229.212:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 08:02:27.521461 31288 ts_manager.cc:194] Registered new tserver with Master: 3ac54d41522441fd83d4e11e6b2b20e4 (127.26.229.196:32795)
I20250624 08:02:27.521997 31288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.196:53025
W20250624 08:02:27.622275 31872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:27.622467 31872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:27.622509 31872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:27.625437 31872 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 08:02:27.625491 31872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:27.625558 31872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.197
I20250624 08:02:27.629009 31872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.26.229.197
--webserver_port=0
--tserver_master_addrs=127.26.229.254:45987
--never_fsync=true
--heap_profile_path=/tmp/kudu.31872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.197
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:27.629931 31872 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:27.631016 31872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:27.638029 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:27.638064 31880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:27.638077 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:27.638280 31872 server_base.cc:1048] running on GCE node
I20250624 08:02:27.638696 31872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:27.639257 31872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:27.640403 31872 hybrid_clock.cc:648] HybridClock initialized: now 1750752147640386 us; error 41 us; skew 500 ppm
I20250624 08:02:27.642149 31872 webserver.cc:469] Webserver started at http://127.26.229.197:43013/ using document root <none> and password file <none>
I20250624 08:02:27.642683 31872 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:27.642740 31872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:27.642968 31872 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:27.644520 31872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6c4fa852063b418c87edae1251299793"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.644971 31872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6c4fa852063b418c87edae1251299793"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.648195 31872 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:27.650113 31886 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.651039 31872 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:27.651135 31872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6c4fa852063b418c87edae1251299793"
format_stamp: "Formatted at 2025-06-24 08:02:27 on dist-test-slave-4j8z"
I20250624 08:02:27.651216 31872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:27.675304 31872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:27.675895 31872 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:27.676051 31872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:27.676573 31872 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:27.677461 31872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:27.677517 31872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.677574 31872 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:27.677601 31872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:27.686703 31872 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.197:37523
I20250624 08:02:27.686758 31999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.197:37523 every 8 connection(s)
I20250624 08:02:27.687610 31872 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250624 08:02:27.695938 32000 heartbeater.cc:344] Connected to a master server at 127.26.229.254:45987
I20250624 08:02:27.696177 32000 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:27.696635 32000 heartbeater.cc:507] Master 127.26.229.254:45987 requested a full tablet report, sending...
I20250624 08:02:27.697117 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 31872
I20250624 08:02:27.697230 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250624 08:02:27.697621 31288 ts_manager.cc:194] Registered new tserver with Master: 6c4fa852063b418c87edae1251299793 (127.26.229.197:37523)
I20250624 08:02:27.698130 31288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.197:57715
I20250624 08:02:27.699728 27543 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250624 08:02:27.707847 31288 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47098:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 08:02:27.724457 31934 tablet_service.cc:1468] Processing CreateTablet for tablet 117e0a3c9d29464babb15acdc83e3764 (DEFAULT_TABLE table=TestTable [id=630d73ea484042d4bae17cae8ff911bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:27.724737 31672 tablet_service.cc:1468] Processing CreateTablet for tablet 117e0a3c9d29464babb15acdc83e3764 (DEFAULT_TABLE table=TestTable [id=630d73ea484042d4bae17cae8ff911bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:27.724737 31541 tablet_service.cc:1468] Processing CreateTablet for tablet 117e0a3c9d29464babb15acdc83e3764 (DEFAULT_TABLE table=TestTable [id=630d73ea484042d4bae17cae8ff911bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:27.725304 31934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 117e0a3c9d29464babb15acdc83e3764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:27.725574 31672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 117e0a3c9d29464babb15acdc83e3764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:27.725574 31541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 117e0a3c9d29464babb15acdc83e3764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:27.729916 32017 tablet_bootstrap.cc:492] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793: Bootstrap starting.
I20250624 08:02:27.731069 32019 tablet_bootstrap.cc:492] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: Bootstrap starting.
I20250624 08:02:27.731297 32018 tablet_bootstrap.cc:492] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a: Bootstrap starting.
I20250624 08:02:27.732219 32017 tablet_bootstrap.cc:654] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:27.733021 32019 tablet_bootstrap.cc:654] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:27.733012 32017 log.cc:826] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:27.733219 32018 tablet_bootstrap.cc:654] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:27.733672 32019 log.cc:826] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:27.733923 32018 log.cc:826] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:27.734653 32017 tablet_bootstrap.cc:492] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793: No bootstrap required, opened a new log
I20250624 08:02:27.734843 32017 ts_tablet_manager.cc:1397] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20250624 08:02:27.735428 32018 tablet_bootstrap.cc:492] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a: No bootstrap required, opened a new log
I20250624 08:02:27.735428 32019 tablet_bootstrap.cc:492] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: No bootstrap required, opened a new log
I20250624 08:02:27.735590 32018 ts_tablet_manager.cc:1397] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 08:02:27.735590 32019 ts_tablet_manager.cc:1397] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250624 08:02:27.736934 32017 raft_consensus.cc:357] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.737108 32017 raft_consensus.cc:383] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:27.737162 32017 raft_consensus.cc:738] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4fa852063b418c87edae1251299793, State: Initialized, Role: FOLLOWER
I20250624 08:02:27.737533 32017 consensus_queue.cc:260] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.738104 32000 heartbeater.cc:499] Master 127.26.229.254:45987 was elected leader, sending a full tablet report...
I20250624 08:02:27.738317 32017 ts_tablet_manager.cc:1428] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250624 08:02:27.738278 32018 raft_consensus.cc:357] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.738456 32018 raft_consensus.cc:383] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:27.738499 32018 raft_consensus.cc:738] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db284502a3b14fffb1ed9472317d632a, State: Initialized, Role: FOLLOWER
I20250624 08:02:27.738891 32018 consensus_queue.cc:260] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.738953 32019 raft_consensus.cc:357] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.739140 32019 raft_consensus.cc:383] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:27.739204 32019 raft_consensus.cc:738] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b493f87f687454cb8c291ed2b01bcd8, State: Initialized, Role: FOLLOWER
I20250624 08:02:27.739555 31607 heartbeater.cc:499] Master 127.26.229.254:45987 was elected leader, sending a full tablet report...
I20250624 08:02:27.739566 32019 consensus_queue.cc:260] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.739673 32018 ts_tablet_manager.cc:1428] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250624 08:02:27.740186 32019 ts_tablet_manager.cc:1428] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250624 08:02:27.740618 31738 heartbeater.cc:499] Master 127.26.229.254:45987 was elected leader, sending a full tablet report...
I20250624 08:02:27.811785 32025 raft_consensus.cc:491] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:27.811980 32025 raft_consensus.cc:513] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.813001 32025 leader_election.cc:290] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db284502a3b14fffb1ed9472317d632a (127.26.229.194:42501), 6c4fa852063b418c87edae1251299793 (127.26.229.197:37523)
I20250624 08:02:27.816656 31561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "117e0a3c9d29464babb15acdc83e3764" candidate_uuid: "8b493f87f687454cb8c291ed2b01bcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db284502a3b14fffb1ed9472317d632a" is_pre_election: true
I20250624 08:02:27.816916 31561 raft_consensus.cc:2466] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b493f87f687454cb8c291ed2b01bcd8 in term 0.
I20250624 08:02:27.817160 31954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "117e0a3c9d29464babb15acdc83e3764" candidate_uuid: "8b493f87f687454cb8c291ed2b01bcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fa852063b418c87edae1251299793" is_pre_election: true
I20250624 08:02:27.817380 31954 raft_consensus.cc:2466] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b493f87f687454cb8c291ed2b01bcd8 in term 0.
I20250624 08:02:27.817385 31625 leader_election.cc:304] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b493f87f687454cb8c291ed2b01bcd8, db284502a3b14fffb1ed9472317d632a; no voters:
I20250624 08:02:27.817623 32025 raft_consensus.cc:2802] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:27.817692 32025 raft_consensus.cc:491] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:27.817749 32025 raft_consensus.cc:3058] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:27.818606 32025 raft_consensus.cc:513] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.819034 32025 leader_election.cc:290] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [CANDIDATE]: Term 1 election: Requested vote from peers db284502a3b14fffb1ed9472317d632a (127.26.229.194:42501), 6c4fa852063b418c87edae1251299793 (127.26.229.197:37523)
I20250624 08:02:27.819324 31561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "117e0a3c9d29464babb15acdc83e3764" candidate_uuid: "8b493f87f687454cb8c291ed2b01bcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db284502a3b14fffb1ed9472317d632a"
I20250624 08:02:27.819383 31954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "117e0a3c9d29464babb15acdc83e3764" candidate_uuid: "8b493f87f687454cb8c291ed2b01bcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4fa852063b418c87edae1251299793"
I20250624 08:02:27.819432 31561 raft_consensus.cc:3058] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:27.819523 31954 raft_consensus.cc:3058] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:27.820322 31561 raft_consensus.cc:2466] T 117e0a3c9d29464babb15acdc83e3764 P db284502a3b14fffb1ed9472317d632a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b493f87f687454cb8c291ed2b01bcd8 in term 1.
I20250624 08:02:27.820780 31954 raft_consensus.cc:2466] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b493f87f687454cb8c291ed2b01bcd8 in term 1.
I20250624 08:02:27.821070 31626 leader_election.cc:304] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4fa852063b418c87edae1251299793, 8b493f87f687454cb8c291ed2b01bcd8; no voters:
I20250624 08:02:27.821233 32025 raft_consensus.cc:2802] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:27.821411 32025 raft_consensus.cc:695] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [term 1 LEADER]: Becoming Leader. State: Replica: 8b493f87f687454cb8c291ed2b01bcd8, State: Running, Role: LEADER
I20250624 08:02:27.821686 32025 consensus_queue.cc:237] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } }
I20250624 08:02:27.824550 31287 catalog_manager.cc:5582] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b493f87f687454cb8c291ed2b01bcd8 (127.26.229.195). New cstate: current_term: 1 leader_uuid: "8b493f87f687454cb8c291ed2b01bcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db284502a3b14fffb1ed9472317d632a" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b493f87f687454cb8c291ed2b01bcd8" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 37955 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:27.837368 27543 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250624 08:02:27.838666 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver db284502a3b14fffb1ed9472317d632a to finish bootstrapping
W20250624 08:02:27.843340 31739 tablet.cc:2378] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:27.844316 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8b493f87f687454cb8c291ed2b01bcd8 to finish bootstrapping
I20250624 08:02:27.848701 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6c4fa852063b418c87edae1251299793 to finish bootstrapping
I20250624 08:02:27.853307 27543 test_util.cc:276] Using random seed: -1076047880
I20250624 08:02:27.861292 27543 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250624 08:02:27.861456 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 31479
W20250624 08:02:27.876376 31625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.229.194:42501: connect: Connection refused (error 111)
I20250624 08:02:27.876825 31954 raft_consensus.cc:1273] T 117e0a3c9d29464babb15acdc83e3764 P 6c4fa852063b418c87edae1251299793 [term 1 FOLLOWER]: Refusing update from remote peer 8b493f87f687454cb8c291ed2b01bcd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250624 08:02:27.877298 31625 consensus_peers.cc:487] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 -> Peer db284502a3b14fffb1ed9472317d632a (127.26.229.194:42501): Couldn't send request to peer db284502a3b14fffb1ed9472317d632a. Status: Network error: Client connection negotiation failed: client connection to 127.26.229.194:42501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 08:02:27.877497 32025 consensus_queue.cc:1035] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4fa852063b418c87edae1251299793" member_type: VOTER last_known_addr { host: "127.26.229.197" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:27.883809 32038 mvcc.cc:204] Tried to move back new op lower bound from 7171080797695782912 to 7171080797480017920. Current Snapshot: MvccSnapshot[applied={T|T < 7171080797695782912}]
I20250624 08:02:27.884500 32040 mvcc.cc:204] Tried to move back new op lower bound from 7171080797695782912 to 7171080797480017920. Current Snapshot: MvccSnapshot[applied={T|T < 7171080797695782912}]
I20250624 08:02:28.021726 31476 heartbeater.cc:499] Master 127.26.229.254:45987 was elected leader, sending a full tablet report...
I20250624 08:02:28.092770 31672 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:28.097756 31803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:28.101478 31934 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:28.103406 31410 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:28.523947 31869 heartbeater.cc:499] Master 127.26.229.254:45987 was elected leader, sending a full tablet report...
I20250624 08:02:29.192667 31410 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:29.193959 31672 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:29.196756 31934 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:29.203879 31803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:30.283535 31410 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:30.283717 31803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:30.285060 31934 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:30.291801 31672 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250624 08:02:30.447175 31625 consensus_peers.cc:487] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 -> Peer db284502a3b14fffb1ed9472317d632a (127.26.229.194:42501): Couldn't send request to peer db284502a3b14fffb1ed9472317d632a. Status: Network error: Client connection negotiation failed: client connection to 127.26.229.194:42501: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250624 08:02:31.372581 31410 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:31.379123 31934 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:31.379590 31672 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:31.389663 31803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:32.468253 31934 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:32.469419 31672 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:32.470191 31803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:32.494261 31410 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250624 08:02:32.937999 31625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.229.194:42501: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250624 08:02:32.938876 31625 consensus_peers.cc:487] T 117e0a3c9d29464babb15acdc83e3764 P 8b493f87f687454cb8c291ed2b01bcd8 -> Peer db284502a3b14fffb1ed9472317d632a (127.26.229.194:42501): Couldn't send request to peer db284502a3b14fffb1ed9472317d632a. Status: Network error: Client connection negotiation failed: client connection to 127.26.229.194:42501: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250624 08:02:33.523049 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 31348
I20250624 08:02:33.529129 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 31610
I20250624 08:02:33.535879 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 31741
I20250624 08:02:33.541360 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 31872
I20250624 08:02:33.547401 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 31257
2025-06-24T08:02:33Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (6982 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (6982 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250624 08:02:33.570662 27543 test_util.cc:276] Using random seed: -1070330532
I20250624 08:02:33.571439 27543 ts_itest-base.cc:115] Starting cluster with:
I20250624 08:02:33.571491 27543 ts_itest-base.cc:116] --------------
I20250624 08:02:33.571503 27543 ts_itest-base.cc:117] 1 tablet servers
I20250624 08:02:33.571514 27543 ts_itest-base.cc:118] 1 replicas per TS
I20250624 08:02:33.571524 27543 ts_itest-base.cc:119] --------------
2025-06-24T08:02:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:33Z Disabled control of system clock
I20250624 08:02:33.580287 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:33177
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:33177 with env {}
W20250624 08:02:33.679730 32223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:33.679977 32223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:33.680044 32223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:33.683125 32223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:33.683177 32223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:33.683209 32223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:33.683241 32223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:33.686915 32223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:33177
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:33.687858 32223 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:33.688875 32223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:33.695151 32228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:33.695155 32231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:33.695250 32223 server_base.cc:1048] running on GCE node
W20250624 08:02:33.695181 32229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:33.695886 32223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:33.696703 32223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:33.697845 32223 hybrid_clock.cc:648] HybridClock initialized: now 1750752153697830 us; error 31 us; skew 500 ppm
I20250624 08:02:33.699612 32223 webserver.cc:469] Webserver started at http://127.26.229.254:40995/ using document root <none> and password file <none>
I20250624 08:02:33.700119 32223 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:33.700176 32223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:33.700337 32223 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:33.702031 32223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3c7a5a53d5a247f2bd67031387f4b0c9"
format_stamp: "Formatted at 2025-06-24 08:02:33 on dist-test-slave-4j8z"
I20250624 08:02:33.702473 32223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3c7a5a53d5a247f2bd67031387f4b0c9"
format_stamp: "Formatted at 2025-06-24 08:02:33 on dist-test-slave-4j8z"
I20250624 08:02:33.705457 32223 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 08:02:33.707244 32237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:33.708192 32223 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250624 08:02:33.708285 32223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3c7a5a53d5a247f2bd67031387f4b0c9"
format_stamp: "Formatted at 2025-06-24 08:02:33 on dist-test-slave-4j8z"
I20250624 08:02:33.708360 32223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:33.731925 32223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:33.732492 32223 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:33.732671 32223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:33.739136 32223 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:33177
I20250624 08:02:33.739149 32289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:33177 every 8 connection(s)
I20250624 08:02:33.740031 32223 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 08:02:33.742413 32290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:33.745560 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 32223
I20250624 08:02:33.745672 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 08:02:33.747530 32290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9: Bootstrap starting.
I20250624 08:02:33.749382 32290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:33.750041 32290 log.cc:826] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:33.751660 32290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9: No bootstrap required, opened a new log
I20250624 08:02:33.754176 32290 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33177 } }
I20250624 08:02:33.754359 32290 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:33.754417 32290 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7a5a53d5a247f2bd67031387f4b0c9, State: Initialized, Role: FOLLOWER
I20250624 08:02:33.754803 32290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33177 } }
I20250624 08:02:33.754956 32290 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:33.755023 32290 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:33.755125 32290 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:33.755822 32290 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33177 } }
I20250624 08:02:33.756110 32290 leader_election.cc:304] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7a5a53d5a247f2bd67031387f4b0c9; no voters:
I20250624 08:02:33.756357 32290 leader_election.cc:290] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:33.756443 32295 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:33.756630 32295 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [term 1 LEADER]: Becoming Leader. State: Replica: 3c7a5a53d5a247f2bd67031387f4b0c9, State: Running, Role: LEADER
I20250624 08:02:33.756930 32295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33177 } }
I20250624 08:02:33.757288 32290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:33.758517 32296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33177 } } }
I20250624 08:02:33.758605 32297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7a5a53d5a247f2bd67031387f4b0c9. Latest consensus state: current_term: 1 leader_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a5a53d5a247f2bd67031387f4b0c9" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 33177 } } }
I20250624 08:02:33.758719 32297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:33.758606 32296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:33.759214 32304 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:33.762228 32304 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:33.766757 32304 catalog_manager.cc:1349] Generated new cluster ID: db094b1c193c4a7bb8fa0885b1bba390
I20250624 08:02:33.766844 32304 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:33.794709 32304 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:33.795961 32304 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:33.814929 32304 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3c7a5a53d5a247f2bd67031387f4b0c9: Generated new TSK 0
I20250624 08:02:33.815583 32304 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:33.820859 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33177
--builtin_ntp_servers=127.26.229.212:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 08:02:33.919848 32314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:33.920037 32314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:33.920095 32314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:33.923254 32314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:33.923350 32314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:33.926869 32314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:33177
--never_fsync=true
--heap_profile_path=/tmp/kudu.32314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:33.927825 32314 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:33.928881 32314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:33.936039 32320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:33.936030 32322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:33.936017 32319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:33.936302 32314 server_base.cc:1048] running on GCE node
I20250624 08:02:33.936616 32314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:33.937143 32314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:33.938297 32314 hybrid_clock.cc:648] HybridClock initialized: now 1750752153938272 us; error 36 us; skew 500 ppm
I20250624 08:02:33.940021 32314 webserver.cc:469] Webserver started at http://127.26.229.193:43807/ using document root <none> and password file <none>
I20250624 08:02:33.940574 32314 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:33.940636 32314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:33.940807 32314 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:33.942298 32314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6d6c6ac7810646b89d953a1622607bbb"
format_stamp: "Formatted at 2025-06-24 08:02:33 on dist-test-slave-4j8z"
I20250624 08:02:33.942759 32314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6d6c6ac7810646b89d953a1622607bbb"
format_stamp: "Formatted at 2025-06-24 08:02:33 on dist-test-slave-4j8z"
I20250624 08:02:33.946005 32314 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:33.948129 32328 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:33.949000 32314 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:33.949100 32314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6d6c6ac7810646b89d953a1622607bbb"
format_stamp: "Formatted at 2025-06-24 08:02:33 on dist-test-slave-4j8z"
I20250624 08:02:33.949200 32314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:33.963948 32314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:33.964536 32314 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:33.964699 32314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:33.965191 32314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:33.966053 32314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:33.966105 32314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:33.966167 32314 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:33.966203 32314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:33.974808 32314 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:33421
I20250624 08:02:33.974858 32441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:33421 every 8 connection(s)
I20250624 08:02:33.975737 32314 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 08:02:33.976572 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 32314
I20250624 08:02:33.976696 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750752133635324-27543-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 08:02:33.984910 32442 heartbeater.cc:344] Connected to a master server at 127.26.229.254:33177
I20250624 08:02:33.985164 32442 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:33.985715 32442 heartbeater.cc:507] Master 127.26.229.254:33177 requested a full tablet report, sending...
I20250624 08:02:33.987196 32254 ts_manager.cc:194] Registered new tserver with Master: 6d6c6ac7810646b89d953a1622607bbb (127.26.229.193:33421)
I20250624 08:02:33.988346 32254 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:42907
I20250624 08:02:33.990936 27543 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250624 08:02:34.000952 32254 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40400:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 08:02:34.015926 32376 tablet_service.cc:1468] Processing CreateTablet for tablet 56d2a0ab078443fd83d89f2d154069dd (DEFAULT_TABLE table=TestTable [id=ecb2fbbd602943a182796808707949ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:34.016783 32376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d2a0ab078443fd83d89f2d154069dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:34.021876 32452 tablet_bootstrap.cc:492] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb: Bootstrap starting.
I20250624 08:02:34.023844 32452 tablet_bootstrap.cc:654] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:34.024487 32452 log.cc:826] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:34.025887 32452 tablet_bootstrap.cc:492] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb: No bootstrap required, opened a new log
I20250624 08:02:34.026069 32452 ts_tablet_manager.cc:1397] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 08:02:34.028308 32452 raft_consensus.cc:357] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6c6ac7810646b89d953a1622607bbb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33421 } }
I20250624 08:02:34.028479 32452 raft_consensus.cc:383] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:34.028563 32452 raft_consensus.cc:738] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6c6ac7810646b89d953a1622607bbb, State: Initialized, Role: FOLLOWER
I20250624 08:02:34.028970 32452 consensus_queue.cc:260] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6c6ac7810646b89d953a1622607bbb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33421 } }
I20250624 08:02:34.029094 32452 raft_consensus.cc:397] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:34.029160 32452 raft_consensus.cc:491] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:34.029265 32452 raft_consensus.cc:3058] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:34.029987 32452 raft_consensus.cc:513] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6c6ac7810646b89d953a1622607bbb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33421 } }
I20250624 08:02:34.030288 32452 leader_election.cc:304] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6c6ac7810646b89d953a1622607bbb; no voters:
I20250624 08:02:34.030567 32452 leader_election.cc:290] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:34.030706 32454 raft_consensus.cc:2802] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:34.030961 32454 raft_consensus.cc:695] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [term 1 LEADER]: Becoming Leader. State: Replica: 6d6c6ac7810646b89d953a1622607bbb, State: Running, Role: LEADER
I20250624 08:02:34.031203 32442 heartbeater.cc:499] Master 127.26.229.254:33177 was elected leader, sending a full tablet report...
I20250624 08:02:34.031394 32452 ts_tablet_manager.cc:1428] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250624 08:02:34.031474 32454 consensus_queue.cc:237] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6c6ac7810646b89d953a1622607bbb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33421 } }
I20250624 08:02:34.034896 32254 catalog_manager.cc:5582] T 56d2a0ab078443fd83d89f2d154069dd P 6d6c6ac7810646b89d953a1622607bbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d6c6ac7810646b89d953a1622607bbb (127.26.229.193). New cstate: current_term: 1 leader_uuid: "6d6c6ac7810646b89d953a1622607bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6c6ac7810646b89d953a1622607bbb" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 33421 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:34.041981 27543 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250624 08:02:34.043321 27543 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6d6c6ac7810646b89d953a1622607bbb to finish bootstrapping
I20250624 08:02:34.189414 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 32314
I20250624 08:02:34.195281 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 32223
2025-06-24T08:02:34Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (645 ms)
[----------] 1 test from ListTableCliSimpleParamTest (645 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250624 08:02:34.216459 27543 test_util.cc:276] Using random seed: -1069684730
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (1 ms)
[----------] 1 test from ListTableCliParamTest (1 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-24T08:02:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T08:02:34Z Disabled control of system clock
I20250624 08:02:34.225725 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:41395
--webserver_interface=127.26.229.254
--webserver_port=0
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:41395 with env {}
W20250624 08:02:34.326903 32476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:34.327100 32476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:34.327154 32476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:34.330080 32476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:34.330125 32476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:34.330152 32476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:34.330169 32476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:34.333634 32476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:41395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:34.334620 32476 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:34.335721 32476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:34.341795 32481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.341795 32484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.341933 32482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:34.342105 32476 server_base.cc:1048] running on GCE node
I20250624 08:02:34.342531 32476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:34.343410 32476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:34.344555 32476 hybrid_clock.cc:648] HybridClock initialized: now 1750752154344535 us; error 39 us; skew 500 ppm
I20250624 08:02:34.346163 32476 webserver.cc:469] Webserver started at http://127.26.229.254:36197/ using document root <none> and password file <none>
I20250624 08:02:34.346657 32476 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:34.346715 32476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:34.346920 32476 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:34.348354 32476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/instance:
uuid: "47a0a9a582cd4447bd224ef9ab812790"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.348799 32476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal/instance:
uuid: "47a0a9a582cd4447bd224ef9ab812790"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.352011 32476 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 08:02:34.354071 32490 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.354983 32476 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:34.355082 32476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
uuid: "47a0a9a582cd4447bd224ef9ab812790"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.355155 32476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:34.382335 32476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:34.383037 32476 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:34.383200 32476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:34.389883 32476 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:41395
I20250624 08:02:34.389895 32542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:41395 every 8 connection(s)
I20250624 08:02:34.390866 32476 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb
I20250624 08:02:34.393270 32543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:34.397974 32543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790: Bootstrap starting.
I20250624 08:02:34.400149 32543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:34.400616 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 32476
I20250624 08:02:34.400713 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal/instance
I20250624 08:02:34.400874 32543 log.cc:826] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:34.402652 32543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790: No bootstrap required, opened a new log
I20250624 08:02:34.404861 32543 raft_consensus.cc:357] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a0a9a582cd4447bd224ef9ab812790" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:34.405037 32543 raft_consensus.cc:383] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:34.405113 32543 raft_consensus.cc:738] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a0a9a582cd4447bd224ef9ab812790, State: Initialized, Role: FOLLOWER
I20250624 08:02:34.405484 32543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a0a9a582cd4447bd224ef9ab812790" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:34.405613 32543 raft_consensus.cc:397] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:34.405720 32543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:34.405838 32543 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:34.406698 32543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a0a9a582cd4447bd224ef9ab812790" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:34.407080 32543 leader_election.cc:304] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a0a9a582cd4447bd224ef9ab812790; no voters:
I20250624 08:02:34.407370 32543 leader_election.cc:290] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:34.407444 32548 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:34.407673 32548 raft_consensus.cc:695] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [term 1 LEADER]: Becoming Leader. State: Replica: 47a0a9a582cd4447bd224ef9ab812790, State: Running, Role: LEADER
I20250624 08:02:34.408512 32543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:34.408797 32548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a0a9a582cd4447bd224ef9ab812790" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:34.410549 32549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a0a9a582cd4447bd224ef9ab812790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a0a9a582cd4447bd224ef9ab812790" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } } }
I20250624 08:02:34.410733 32550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a0a9a582cd4447bd224ef9ab812790. Latest consensus state: current_term: 1 leader_uuid: "47a0a9a582cd4447bd224ef9ab812790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a0a9a582cd4447bd224ef9ab812790" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } } }
I20250624 08:02:34.411063 32549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:34.411082 32550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790 [sys.catalog]: This master's current role is: LEADER
W20250624 08:02:34.413069 32562 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250624 08:02:34.413133 32562 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250624 08:02:34.413245 32563 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:34.414188 32563 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:34.419214 32563 catalog_manager.cc:1349] Generated new cluster ID: 4105788b3f254e2a8bbe634e482dccd5
I20250624 08:02:34.419286 32563 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:34.446514 32563 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:34.447523 32563 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:34.456182 32563 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 47a0a9a582cd4447bd224ef9ab812790: Generated new TSK 0
I20250624 08:02:34.456895 32563 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 08:02:34.463977 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:0
--local_ip_for_outbound_sockets=127.26.229.193
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:41395
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250624 08:02:34.568040 32567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:34.568254 32567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:34.568296 32567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:34.571506 32567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:34.571604 32567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:34.575212 32567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=0
--tserver_master_addrs=127.26.229.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.32567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:34.576193 32567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:34.577299 32567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:34.584689 32573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.584779 32572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.584863 32575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:34.584923 32567 server_base.cc:1048] running on GCE node
I20250624 08:02:34.585350 32567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:34.585896 32567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:34.587040 32567 hybrid_clock.cc:648] HybridClock initialized: now 1750752154587025 us; error 29 us; skew 500 ppm
I20250624 08:02:34.588757 32567 webserver.cc:469] Webserver started at http://127.26.229.193:35281/ using document root <none> and password file <none>
I20250624 08:02:34.589296 32567 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:34.589351 32567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:34.589537 32567 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:34.591080 32567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/instance:
uuid: "5dfd251ca84340fc923e0da615d3252c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.591476 32567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal/instance:
uuid: "5dfd251ca84340fc923e0da615d3252c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.594523 32567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250624 08:02:34.596537 32581 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.597481 32567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250624 08:02:34.597579 32567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
uuid: "5dfd251ca84340fc923e0da615d3252c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.597678 32567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:34.608001 32567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:34.608561 32567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:34.608716 32567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:34.609210 32567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:34.610174 32567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:34.610219 32567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.610261 32567 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:34.610294 32567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.618577 32567 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:38463
I20250624 08:02:34.618611 32694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:38463 every 8 connection(s)
I20250624 08:02:34.619467 32567 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb
I20250624 08:02:34.620050 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 32567
I20250624 08:02:34.620195 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal/instance
I20250624 08:02:34.622334 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:0
--local_ip_for_outbound_sockets=127.26.229.194
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:41395
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:02:34.628582 32695 heartbeater.cc:344] Connected to a master server at 127.26.229.254:41395
I20250624 08:02:34.628831 32695 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:34.629303 32695 heartbeater.cc:507] Master 127.26.229.254:41395 requested a full tablet report, sending...
I20250624 08:02:34.630738 32506 ts_manager.cc:194] Registered new tserver with Master: 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463)
I20250624 08:02:34.632422 32506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:45325
W20250624 08:02:34.720978 32698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:34.721227 32698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:34.721282 32698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:34.724292 32698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:34.724381 32698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:34.727770 32698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=0
--tserver_master_addrs=127.26.229.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.32698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:34.728657 32698 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:34.729672 32698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:34.736631 32704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.736748 32703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.736753 32706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:34.736953 32698 server_base.cc:1048] running on GCE node
I20250624 08:02:34.737290 32698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:34.737880 32698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:34.739049 32698 hybrid_clock.cc:648] HybridClock initialized: now 1750752154739028 us; error 45 us; skew 500 ppm
I20250624 08:02:34.740707 32698 webserver.cc:469] Webserver started at http://127.26.229.194:42099/ using document root <none> and password file <none>
I20250624 08:02:34.741202 32698 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:34.741255 32698 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:34.741431 32698 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:34.742949 32698 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/instance:
uuid: "3c4884b8e5b74b6da18da40034a6737c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.743362 32698 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal/instance:
uuid: "3c4884b8e5b74b6da18da40034a6737c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.746387 32698 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:34.748411 32712 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.749403 32698 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:34.749488 32698 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
uuid: "3c4884b8e5b74b6da18da40034a6737c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.749588 32698 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:34.762135 32698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:34.762650 32698 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:34.762861 32698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:34.763343 32698 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:34.764148 32698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:34.764191 32698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.764243 32698 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:34.764277 32698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.772152 32698 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:36983
I20250624 08:02:34.772181 357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:36983 every 8 connection(s)
I20250624 08:02:34.772984 32698 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb
I20250624 08:02:34.778167 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 32698
I20250624 08:02:34.778297 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal/instance
I20250624 08:02:34.780473 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:0
--local_ip_for_outbound_sockets=127.26.229.195
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:41395
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:02:34.781205 358 heartbeater.cc:344] Connected to a master server at 127.26.229.254:41395
I20250624 08:02:34.781432 358 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:34.782025 358 heartbeater.cc:507] Master 127.26.229.254:41395 requested a full tablet report, sending...
I20250624 08:02:34.782953 32506 ts_manager.cc:194] Registered new tserver with Master: 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983)
I20250624 08:02:34.783700 32506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:47217
W20250624 08:02:34.875897 361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:34.876078 361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:34.876117 361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:34.878926 361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:34.879009 361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:34.882405 361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=0
--tserver_master_addrs=127.26.229.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:34.883312 361 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:34.884289 361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:34.891445 369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.891518 367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:34.891434 366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:34.891691 361 server_base.cc:1048] running on GCE node
I20250624 08:02:34.892047 361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:34.892561 361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:34.893707 361 hybrid_clock.cc:648] HybridClock initialized: now 1750752154893683 us; error 38 us; skew 500 ppm
I20250624 08:02:34.895298 361 webserver.cc:469] Webserver started at http://127.26.229.195:34085/ using document root <none> and password file <none>
I20250624 08:02:34.895880 361 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:34.895941 361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:34.896113 361 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:34.897612 361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/instance:
uuid: "b20f7081883143e38d3713f88507ac8f"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.898036 361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal/instance:
uuid: "b20f7081883143e38d3713f88507ac8f"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.901163 361 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:34.903172 375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.904008 361 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:34.904099 361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
uuid: "b20f7081883143e38d3713f88507ac8f"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:34.904175 361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:34.916146 361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:34.916653 361 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:34.916800 361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:34.917268 361 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:34.918052 361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 08:02:34.918102 361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.918155 361 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 08:02:34.918182 361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:34.926573 361 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:45975
I20250624 08:02:34.926594 488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:45975 every 8 connection(s)
I20250624 08:02:34.927397 361 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb
I20250624 08:02:34.935155 489 heartbeater.cc:344] Connected to a master server at 127.26.229.254:41395
I20250624 08:02:34.935381 489 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:34.935853 489 heartbeater.cc:507] Master 127.26.229.254:41395 requested a full tablet report, sending...
I20250624 08:02:34.936307 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 361
I20250624 08:02:34.936416 27543 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal/instance
I20250624 08:02:34.936866 32506 ts_manager.cc:194] Registered new tserver with Master: b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975)
I20250624 08:02:34.937471 32506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:44511
I20250624 08:02:34.938457 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:34.944960 27543 test_util.cc:276] Using random seed: -1068956226
I20250624 08:02:34.953502 32506 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34656:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 08:02:34.954564 32506 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:02:34.968900 423 tablet_service.cc:1468] Processing CreateTablet for tablet c53ca2985a1c4669a6cc8000a67d8008 (DEFAULT_TABLE table=pre_rebuild [id=2ca1ea1dc19d4014a23672d0e6280f7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:34.968904 32760 tablet_service.cc:1468] Processing CreateTablet for tablet c53ca2985a1c4669a6cc8000a67d8008 (DEFAULT_TABLE table=pre_rebuild [id=2ca1ea1dc19d4014a23672d0e6280f7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:34.968901 32629 tablet_service.cc:1468] Processing CreateTablet for tablet c53ca2985a1c4669a6cc8000a67d8008 (DEFAULT_TABLE table=pre_rebuild [id=2ca1ea1dc19d4014a23672d0e6280f7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:34.969797 423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53ca2985a1c4669a6cc8000a67d8008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:34.969808 32760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53ca2985a1c4669a6cc8000a67d8008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:34.969808 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53ca2985a1c4669a6cc8000a67d8008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:34.974470 511 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Bootstrap starting.
I20250624 08:02:34.975881 512 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Bootstrap starting.
I20250624 08:02:34.976478 511 tablet_bootstrap.cc:654] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:34.977304 511 log.cc:826] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:34.977662 512 tablet_bootstrap.cc:654] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:34.978024 513 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Bootstrap starting.
I20250624 08:02:34.978392 512 log.cc:826] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:34.979036 511 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: No bootstrap required, opened a new log
I20250624 08:02:34.979256 511 ts_tablet_manager.cc:1397] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250624 08:02:34.979790 513 tablet_bootstrap.cc:654] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:34.980067 512 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: No bootstrap required, opened a new log
I20250624 08:02:34.980239 512 ts_tablet_manager.cc:1397] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 08:02:34.980428 513 log.cc:826] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:34.981714 513 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: No bootstrap required, opened a new log
I20250624 08:02:34.981877 513 ts_tablet_manager.cc:1397] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250624 08:02:34.982234 511 raft_consensus.cc:357] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:34.982443 511 raft_consensus.cc:383] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:34.982717 511 raft_consensus.cc:738] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20f7081883143e38d3713f88507ac8f, State: Initialized, Role: FOLLOWER
I20250624 08:02:34.983274 511 consensus_queue.cc:260] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:34.983283 512 raft_consensus.cc:357] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:34.983464 512 raft_consensus.cc:383] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:34.983570 512 raft_consensus.cc:738] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4884b8e5b74b6da18da40034a6737c, State: Initialized, Role: FOLLOWER
I20250624 08:02:34.984023 512 consensus_queue.cc:260] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:34.984145 511 ts_tablet_manager.cc:1428] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20250624 08:02:34.984485 489 heartbeater.cc:499] Master 127.26.229.254:41395 was elected leader, sending a full tablet report...
I20250624 08:02:34.984448 513 raft_consensus.cc:357] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:34.984627 513 raft_consensus.cc:383] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:34.984664 513 raft_consensus.cc:738] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfd251ca84340fc923e0da615d3252c, State: Initialized, Role: FOLLOWER
I20250624 08:02:34.984666 358 heartbeater.cc:499] Master 127.26.229.254:41395 was elected leader, sending a full tablet report...
I20250624 08:02:34.985009 513 consensus_queue.cc:260] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:34.985541 32695 heartbeater.cc:499] Master 127.26.229.254:41395 was elected leader, sending a full tablet report...
I20250624 08:02:34.985730 513 ts_tablet_manager.cc:1428] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20250624 08:02:34.986245 512 ts_tablet_manager.cc:1428] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250624 08:02:35.023541 517 raft_consensus.cc:491] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250624 08:02:35.023772 359 tablet.cc:2378] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:35.023762 517 raft_consensus.cc:513] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:35.024755 517 leader_election.cc:290] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463), 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983)
I20250624 08:02:35.028112 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "b20f7081883143e38d3713f88507ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dfd251ca84340fc923e0da615d3252c" is_pre_election: true
I20250624 08:02:35.028357 32649 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20f7081883143e38d3713f88507ac8f in term 0.
I20250624 08:02:35.028797 379 leader_election.cc:304] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfd251ca84340fc923e0da615d3252c, b20f7081883143e38d3713f88507ac8f; no voters:
I20250624 08:02:35.028908 312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "b20f7081883143e38d3713f88507ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4884b8e5b74b6da18da40034a6737c" is_pre_election: true
I20250624 08:02:35.028992 517 raft_consensus.cc:2802] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:35.029065 517 raft_consensus.cc:491] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:35.029114 517 raft_consensus.cc:3058] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:35.029157 312 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20f7081883143e38d3713f88507ac8f in term 0.
I20250624 08:02:35.030098 517 raft_consensus.cc:513] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:35.030371 517 leader_election.cc:290] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [CANDIDATE]: Term 1 election: Requested vote from peers 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463), 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983)
I20250624 08:02:35.030709 312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "b20f7081883143e38d3713f88507ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4884b8e5b74b6da18da40034a6737c"
I20250624 08:02:35.030709 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "b20f7081883143e38d3713f88507ac8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dfd251ca84340fc923e0da615d3252c"
I20250624 08:02:35.030838 312 raft_consensus.cc:3058] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:35.030844 32649 raft_consensus.cc:3058] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:35.031978 312 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20f7081883143e38d3713f88507ac8f in term 1.
I20250624 08:02:35.032298 376 leader_election.cc:304] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4884b8e5b74b6da18da40034a6737c, b20f7081883143e38d3713f88507ac8f; no voters:
I20250624 08:02:35.032385 32649 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20f7081883143e38d3713f88507ac8f in term 1.
I20250624 08:02:35.032487 517 raft_consensus.cc:2802] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:35.032696 517 raft_consensus.cc:695] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 LEADER]: Becoming Leader. State: Replica: b20f7081883143e38d3713f88507ac8f, State: Running, Role: LEADER
I20250624 08:02:35.032991 517 consensus_queue.cc:237] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:35.035878 32506 catalog_manager.cc:5582] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f reported cstate change: term changed from 0 to 1, leader changed from <none> to b20f7081883143e38d3713f88507ac8f (127.26.229.195). New cstate: current_term: 1 leader_uuid: "b20f7081883143e38d3713f88507ac8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20250624 08:02:35.078644 32649 raft_consensus.cc:1273] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Refusing update from remote peer b20f7081883143e38d3713f88507ac8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:02:35.078644 312 raft_consensus.cc:1273] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Refusing update from remote peer b20f7081883143e38d3713f88507ac8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:02:35.079326 517 consensus_queue.cc:1035] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:35.079478 521 consensus_queue.cc:1035] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:35.088202 531 mvcc.cc:204] Tried to move back new op lower bound from 7171080827206443008 to 7171080827017543680. Current Snapshot: MvccSnapshot[applied={T|T < 7171080827194925056}]
I20250624 08:02:35.907755 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 32476
W20250624 08:02:36.037850 551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:36.038036 551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:36.041162 551 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 08:02:36.071475 551 minidump.cc:252] Setting minidump size limit to 20M
I20250624 08:02:36.072433 551 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:36.073215 551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:36.077389 574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.077543 575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.077389 577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:36.077685 551 server_base.cc:1048] running on GCE node
I20250624 08:02:36.078148 551 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250624 08:02:36.078323 551 hybrid_clock.cc:648] HybridClock initialized: now 1750752156078318 us; error 68590 us; skew 500 ppm
I20250624 08:02:36.079864 551 webserver.cc:469] Webserver started at http://0.0.0.0:43375/ using document root <none> and password file <none>
I20250624 08:02:36.080426 551 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:36.080493 551 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:36.080641 551 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 08:02:36.082056 551 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/instance:
uuid: "5e248061ebd6479ca128dd82ead7cc55"
format_stamp: "Formatted at 2025-06-24 08:02:36 on dist-test-slave-4j8z"
I20250624 08:02:36.082484 551 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal/instance:
uuid: "5e248061ebd6479ca128dd82ead7cc55"
format_stamp: "Formatted at 2025-06-24 08:02:36 on dist-test-slave-4j8z"
I20250624 08:02:36.085477 551 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 08:02:36.087386 582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:36.088068 551 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250624 08:02:36.088157 551 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
uuid: "5e248061ebd6479ca128dd82ead7cc55"
format_stamp: "Formatted at 2025-06-24 08:02:36 on dist-test-slave-4j8z"
I20250624 08:02:36.088222 551 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250624 08:02:36.091233 358 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:41395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:41395: connect: Connection refused (error 111)
W20250624 08:02:36.096222 32695 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:41395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:41395: connect: Connection refused (error 111)
W20250624 08:02:36.101243 489 heartbeater.cc:646] Failed to heartbeat to 127.26.229.254:41395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.229.254:41395: connect: Connection refused (error 111)
I20250624 08:02:36.351449 551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:36.352021 551 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:36.352157 551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:36.354137 551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:36.358932 551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Bootstrap starting.
I20250624 08:02:36.360869 551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:36.361838 551 log.cc:826] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:36.363565 551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: No bootstrap required, opened a new log
I20250624 08:02:36.365768 551 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER }
I20250624 08:02:36.365923 551 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:36.365960 551 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e248061ebd6479ca128dd82ead7cc55, State: Initialized, Role: FOLLOWER
I20250624 08:02:36.366304 551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER }
I20250624 08:02:36.366394 551 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:36.366442 551 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:36.366516 551 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:36.367206 551 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER }
I20250624 08:02:36.367508 551 leader_election.cc:304] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e248061ebd6479ca128dd82ead7cc55; no voters:
I20250624 08:02:36.367765 551 leader_election.cc:290] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 08:02:36.367868 593 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:36.368062 593 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 LEADER]: Becoming Leader. State: Replica: 5e248061ebd6479ca128dd82ead7cc55, State: Running, Role: LEADER
I20250624 08:02:36.368383 593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER }
I20250624 08:02:36.369827 595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e248061ebd6479ca128dd82ead7cc55. Latest consensus state: current_term: 1 leader_uuid: "5e248061ebd6479ca128dd82ead7cc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER } }
I20250624 08:02:36.369933 594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e248061ebd6479ca128dd82ead7cc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER } }
I20250624 08:02:36.370051 594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:36.370051 595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:36.374573 551 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: stopping tablet replica
I20250624 08:02:36.374836 551 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 LEADER]: Raft consensus shutting down.
I20250624 08:02:36.374956 551 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 08:02:36.375854 551 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250624 08:02:36.376083 551 acceptor_pool.cc:194] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250624 08:02:36.387202 551 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250624 08:02:36.396957 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 32567
I20250624 08:02:36.402817 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 32698
I20250624 08:02:36.408028 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 361
I20250624 08:02:36.414369 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:41395
--webserver_interface=127.26.229.254
--webserver_port=36197
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.229.254:41395 with env {}
W20250624 08:02:36.511572 599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:36.511803 599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:36.511873 599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:36.514777 599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 08:02:36.514858 599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:36.514889 599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 08:02:36.514922 599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 08:02:36.518263 599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.229.254:41395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.229.254:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.229.254
--webserver_port=36197
--never_fsync=true
--heap_profile_path=/tmp/kudu.599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:36.519269 599 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:36.520345 599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:36.526620 604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.526757 605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:36.526877 599 server_base.cc:1048] running on GCE node
W20250624 08:02:36.526991 607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:36.527413 599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:36.528255 599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:36.529405 599 hybrid_clock.cc:648] HybridClock initialized: now 1750752156529392 us; error 31 us; skew 500 ppm
I20250624 08:02:36.531158 599 webserver.cc:469] Webserver started at http://127.26.229.254:36197/ using document root <none> and password file <none>
I20250624 08:02:36.531726 599 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:36.531798 599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:36.535001 599 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20250624 08:02:36.536806 613 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:36.537748 599 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:36.537865 599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
uuid: "5e248061ebd6479ca128dd82ead7cc55"
format_stamp: "Formatted at 2025-06-24 08:02:36 on dist-test-slave-4j8z"
I20250624 08:02:36.538221 599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:36.554692 599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:36.555316 599 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:36.555483 599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:36.562691 599 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.254:41395
I20250624 08:02:36.562726 665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.254:41395 every 8 connection(s)
I20250624 08:02:36.563812 599 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb
I20250624 08:02:36.567255 666 sys_catalog.cc:263] Verifying existing consensus state
I20250624 08:02:36.569418 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 599
I20250624 08:02:36.569907 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.193:38463
--local_ip_for_outbound_sockets=127.26.229.193
--tserver_master_addrs=127.26.229.254:41395
--webserver_port=35281
--webserver_interface=127.26.229.193
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:02:36.571110 666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Bootstrap starting.
I20250624 08:02:36.575366 666 log.cc:826] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:36.580042 666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:36.580389 666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Bootstrap complete.
I20250624 08:02:36.582899 666 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:36.583065 666 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e248061ebd6479ca128dd82ead7cc55, State: Initialized, Role: FOLLOWER
I20250624 08:02:36.583498 666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:36.583611 666 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 08:02:36.583673 666 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 08:02:36.583765 666 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:36.584419 666 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:36.584676 666 leader_election.cc:304] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e248061ebd6479ca128dd82ead7cc55; no voters:
I20250624 08:02:36.584892 666 leader_election.cc:290] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 08:02:36.584970 670 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:36.585122 670 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [term 2 LEADER]: Becoming Leader. State: Replica: 5e248061ebd6479ca128dd82ead7cc55, State: Running, Role: LEADER
I20250624 08:02:36.585403 670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } }
I20250624 08:02:36.585913 666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 08:02:36.586249 670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e248061ebd6479ca128dd82ead7cc55. Latest consensus state: current_term: 2 leader_uuid: "5e248061ebd6479ca128dd82ead7cc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } } }
I20250624 08:02:36.586349 670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:36.586670 674 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 08:02:36.586686 671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e248061ebd6479ca128dd82ead7cc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e248061ebd6479ca128dd82ead7cc55" member_type: VOTER last_known_addr { host: "127.26.229.254" port: 41395 } } }
I20250624 08:02:36.586892 671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55 [sys.catalog]: This master's current role is: LEADER
I20250624 08:02:36.590165 674 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=5a92098ee0d34656834ebd9c4ec16a7c]
I20250624 08:02:36.591380 674 tablet_loader.cc:96] loaded metadata for tablet c53ca2985a1c4669a6cc8000a67d8008 (table pre_rebuild [id=5a92098ee0d34656834ebd9c4ec16a7c])
I20250624 08:02:36.591516 674 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 08:02:36.595062 674 catalog_manager.cc:1349] Generated new cluster ID: 29ed9299215e4be4985f5a73025b719c
I20250624 08:02:36.595116 674 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 08:02:36.612422 674 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 08:02:36.613459 674 catalog_manager.cc:1506] Loading token signing keys...
I20250624 08:02:36.623872 674 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Generated new TSK 0
I20250624 08:02:36.624518 674 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 08:02:36.670847 668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:36.671054 668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:36.671123 668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:36.674227 668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:36.674347 668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.193
I20250624 08:02:36.677743 668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.193:38463
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.229.193
--webserver_port=35281
--tserver_master_addrs=127.26.229.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.193
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:36.678666 668 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:36.679708 668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:36.687218 696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.687209 694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.687539 693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:36.687562 668 server_base.cc:1048] running on GCE node
I20250624 08:02:36.687973 668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:36.688544 668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:36.689700 668 hybrid_clock.cc:648] HybridClock initialized: now 1750752156689682 us; error 35 us; skew 500 ppm
I20250624 08:02:36.691696 668 webserver.cc:469] Webserver started at http://127.26.229.193:35281/ using document root <none> and password file <none>
I20250624 08:02:36.692337 668 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:36.692425 668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:36.696426 668 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20250624 08:02:36.698442 702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:36.699501 668 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250624 08:02:36.699604 668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
uuid: "5dfd251ca84340fc923e0da615d3252c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:36.700035 668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:36.716542 668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:36.717154 668 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:36.717314 668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:36.717921 668 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:36.719153 709 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250624 08:02:36.722036 668 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:02:36.722105 668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250624 08:02:36.722163 668 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:02:36.723919 668 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:02:36.723973 668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:36.724299 709 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Bootstrap starting.
I20250624 08:02:36.735200 668 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.193:38463
I20250624 08:02:36.735384 816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.193:38463 every 8 connection(s)
I20250624 08:02:36.736356 668 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb
I20250624 08:02:36.745766 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 668
I20250624 08:02:36.746210 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.194:36983
--local_ip_for_outbound_sockets=127.26.229.194
--tserver_master_addrs=127.26.229.254:41395
--webserver_port=42099
--webserver_interface=127.26.229.194
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:02:36.749128 817 heartbeater.cc:344] Connected to a master server at 127.26.229.254:41395
I20250624 08:02:36.749357 817 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:36.749948 817 heartbeater.cc:507] Master 127.26.229.254:41395 requested a full tablet report, sending...
I20250624 08:02:36.752538 630 ts_manager.cc:194] Registered new tserver with Master: 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463)
I20250624 08:02:36.755065 630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.193:45169
I20250624 08:02:36.761366 709 log.cc:826] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Log is configured to *not* fsync() on all Append() calls
W20250624 08:02:36.878088 821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:36.878304 821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:36.878377 821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:36.881392 821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:36.881515 821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.194
I20250624 08:02:36.885001 821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.194:36983
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.229.194
--webserver_port=42099
--tserver_master_addrs=127.26.229.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.194
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:36.886006 821 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:36.887156 821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:36.895130 827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.895175 828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:36.895213 830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:36.895692 821 server_base.cc:1048] running on GCE node
I20250624 08:02:36.896016 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:36.896574 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:36.897737 821 hybrid_clock.cc:648] HybridClock initialized: now 1750752156897728 us; error 43 us; skew 500 ppm
I20250624 08:02:36.899717 821 webserver.cc:469] Webserver started at http://127.26.229.194:42099/ using document root <none> and password file <none>
I20250624 08:02:36.900295 821 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:36.900388 821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:36.903613 821 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250624 08:02:36.905532 836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:36.906488 821 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:36.906579 821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
uuid: "3c4884b8e5b74b6da18da40034a6737c"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:36.906982 821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:36.936064 821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:36.936699 821 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:36.936865 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:36.937417 821 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:36.938617 843 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250624 08:02:36.941728 821 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:02:36.941785 821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.001s sys 0.000s
I20250624 08:02:36.941840 821 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:02:36.943324 821 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:02:36.943394 821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:36.943701 843 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Bootstrap starting.
I20250624 08:02:36.955134 821 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.194:36983
I20250624 08:02:36.956337 821 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb
I20250624 08:02:36.957368 950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.194:36983 every 8 connection(s)
I20250624 08:02:36.959455 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 821
I20250624 08:02:36.959895 27543 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7veRrf/build/debug/bin/kudu
/tmp/dist-test-task7veRrf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.229.195:45975
--local_ip_for_outbound_sockets=127.26.229.195
--tserver_master_addrs=127.26.229.254:41395
--webserver_port=34085
--webserver_interface=127.26.229.195
--builtin_ntp_servers=127.26.229.212:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 08:02:36.969616 951 heartbeater.cc:344] Connected to a master server at 127.26.229.254:41395
I20250624 08:02:36.969851 951 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:36.970405 951 heartbeater.cc:507] Master 127.26.229.254:41395 requested a full tablet report, sending...
I20250624 08:02:36.972142 630 ts_manager.cc:194] Registered new tserver with Master: 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983)
I20250624 08:02:36.973169 630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.194:34275
I20250624 08:02:36.990731 843 log.cc:826] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:37.071609 709 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:37.072021 709 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Bootstrap complete.
I20250624 08:02:37.073093 709 ts_tablet_manager.cc:1397] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Time spent bootstrapping tablet: real 0.349s user 0.318s sys 0.028s
I20250624 08:02:37.075042 709 raft_consensus.cc:357] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:37.075358 709 raft_consensus.cc:738] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfd251ca84340fc923e0da615d3252c, State: Initialized, Role: FOLLOWER
I20250624 08:02:37.075814 709 consensus_queue.cc:260] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:37.076532 817 heartbeater.cc:499] Master 127.26.229.254:41395 was elected leader, sending a full tablet report...
I20250624 08:02:37.076647 709 ts_tablet_manager.cc:1428] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20250624 08:02:37.100921 954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 08:02:37.101178 954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 08:02:37.101244 954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 08:02:37.105011 954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 08:02:37.105142 954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.229.195
I20250624 08:02:37.109318 954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.229.212:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.229.195:45975
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.229.195
--webserver_port=34085
--tserver_master_addrs=127.26.229.254:41395
--never_fsync=true
--heap_profile_path=/tmp/kudu.954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.229.195
--log_dir=/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 24 Jun 2025 07:43:12 UTC on 24a791456cd2
build id 6742
I20250624 08:02:37.110416 954 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 08:02:37.111722 954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 08:02:37.120146 964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 08:02:37.120364 962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:37.120638 954 server_base.cc:1048] running on GCE node
W20250624 08:02:37.120800 961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 08:02:37.121145 954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 08:02:37.121644 954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 08:02:37.122807 954 hybrid_clock.cc:648] HybridClock initialized: now 1750752157122789 us; error 35 us; skew 500 ppm
I20250624 08:02:37.124749 954 webserver.cc:469] Webserver started at http://127.26.229.195:34085/ using document root <none> and password file <none>
I20250624 08:02:37.125443 954 fs_manager.cc:362] Metadata directory not provided
I20250624 08:02:37.125530 954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 08:02:37.129690 954 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:37.131948 970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 08:02:37.133116 954 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250624 08:02:37.133229 954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data,/tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
uuid: "b20f7081883143e38d3713f88507ac8f"
format_stamp: "Formatted at 2025-06-24 08:02:34 on dist-test-slave-4j8z"
I20250624 08:02:37.133682 954 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 08:02:37.166756 954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 08:02:37.167562 954 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 08:02:37.167752 954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 08:02:37.168424 954 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 08:02:37.169895 977 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250624 08:02:37.173187 954 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 08:02:37.173256 954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250624 08:02:37.173316 954 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 08:02:37.175134 954 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 08:02:37.175189 954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20250624 08:02:37.175479 977 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Bootstrap starting.
I20250624 08:02:37.186285 954 rpc_server.cc:307] RPC server started. Bound to: 127.26.229.195:45975
I20250624 08:02:37.186405 1084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.229.195:45975 every 8 connection(s)
I20250624 08:02:37.187575 954 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb
I20250624 08:02:37.188596 27543 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7veRrf/build/debug/bin/kudu as pid 954
I20250624 08:02:37.198279 1085 heartbeater.cc:344] Connected to a master server at 127.26.229.254:41395
I20250624 08:02:37.198524 1085 heartbeater.cc:461] Registering TS with master...
I20250624 08:02:37.199160 1085 heartbeater.cc:507] Master 127.26.229.254:41395 requested a full tablet report, sending...
I20250624 08:02:37.201000 630 ts_manager.cc:194] Registered new tserver with Master: b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975)
I20250624 08:02:37.201912 630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.229.195:59823
I20250624 08:02:37.210150 27543 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 08:02:37.214215 977 log.cc:826] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Log is configured to *not* fsync() on all Append() calls
I20250624 08:02:37.372689 843 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:37.373097 843 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Bootstrap complete.
I20250624 08:02:37.374150 843 ts_tablet_manager.cc:1397] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Time spent bootstrapping tablet: real 0.431s user 0.399s sys 0.028s
I20250624 08:02:37.376199 843 raft_consensus.cc:357] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:37.376513 843 raft_consensus.cc:738] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4884b8e5b74b6da18da40034a6737c, State: Initialized, Role: FOLLOWER
I20250624 08:02:37.376942 843 consensus_queue.cc:260] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:37.377656 951 heartbeater.cc:499] Master 127.26.229.254:41395 was elected leader, sending a full tablet report...
I20250624 08:02:37.377769 843 ts_tablet_manager.cc:1428] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20250624 08:02:37.528451 977 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 08:02:37.528863 977 tablet_bootstrap.cc:492] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Bootstrap complete.
I20250624 08:02:37.529963 977 ts_tablet_manager.cc:1397] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Time spent bootstrapping tablet: real 0.355s user 0.346s sys 0.004s
I20250624 08:02:37.531987 977 raft_consensus.cc:357] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:37.532300 977 raft_consensus.cc:738] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20f7081883143e38d3713f88507ac8f, State: Initialized, Role: FOLLOWER
I20250624 08:02:37.532729 977 consensus_queue.cc:260] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:37.533349 1085 heartbeater.cc:499] Master 127.26.229.254:41395 was elected leader, sending a full tablet report...
I20250624 08:02:37.533488 977 ts_tablet_manager.cc:1428] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20250624 08:02:38.584182 1098 raft_consensus.cc:491] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:38.584385 1098 raft_consensus.cc:513] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:38.585475 1098 leader_election.cc:290] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463), b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975)
I20250624 08:02:38.590416 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "3c4884b8e5b74b6da18da40034a6737c" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b20f7081883143e38d3713f88507ac8f" is_pre_election: true
I20250624 08:02:38.590407 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "3c4884b8e5b74b6da18da40034a6737c" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "5dfd251ca84340fc923e0da615d3252c" is_pre_election: true
I20250624 08:02:38.590682 771 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4884b8e5b74b6da18da40034a6737c in term 1.
I20250624 08:02:38.590682 1039 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4884b8e5b74b6da18da40034a6737c in term 1.
I20250624 08:02:38.591423 837 leader_election.cc:304] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4884b8e5b74b6da18da40034a6737c, b20f7081883143e38d3713f88507ac8f; no voters:
I20250624 08:02:38.591660 1098 raft_consensus.cc:2802] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 08:02:38.591733 1098 raft_consensus.cc:491] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:38.591776 1098 raft_consensus.cc:3058] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:38.681555 1098 raft_consensus.cc:513] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:38.682106 1098 leader_election.cc:290] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [CANDIDATE]: Term 2 election: Requested vote from peers 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463), b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975)
I20250624 08:02:38.682667 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "3c4884b8e5b74b6da18da40034a6737c" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "5dfd251ca84340fc923e0da615d3252c"
I20250624 08:02:38.682685 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c53ca2985a1c4669a6cc8000a67d8008" candidate_uuid: "3c4884b8e5b74b6da18da40034a6737c" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b20f7081883143e38d3713f88507ac8f"
I20250624 08:02:38.682844 771 raft_consensus.cc:3058] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:38.682876 1039 raft_consensus.cc:3058] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Advancing to term 2
I20250624 08:02:38.684614 771 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4884b8e5b74b6da18da40034a6737c in term 2.
I20250624 08:02:38.685011 840 leader_election.cc:304] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4884b8e5b74b6da18da40034a6737c, 5dfd251ca84340fc923e0da615d3252c; no voters:
I20250624 08:02:38.685220 1098 raft_consensus.cc:2802] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 FOLLOWER]: Leader election won for term 2
I20250624 08:02:38.685273 1039 raft_consensus.cc:2466] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4884b8e5b74b6da18da40034a6737c in term 2.
I20250624 08:02:38.685424 1098 raft_consensus.cc:695] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 LEADER]: Becoming Leader. State: Replica: 3c4884b8e5b74b6da18da40034a6737c, State: Running, Role: LEADER
I20250624 08:02:38.685768 1098 consensus_queue.cc:237] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:38.688624 630 catalog_manager.cc:5582] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c reported cstate change: term changed from 0 to 2, leader changed from <none> to 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194), VOTER 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194) added, VOTER 5dfd251ca84340fc923e0da615d3252c (127.26.229.193) added, VOTER b20f7081883143e38d3713f88507ac8f (127.26.229.195) added. New cstate: current_term: 2 leader_uuid: "3c4884b8e5b74b6da18da40034a6737c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } health_report { overall_health: HEALTHY } } }
W20250624 08:02:38.764467 27543 scanner-internal.cc:458] Time spent opening tablet: real 1.542s user 0.003s sys 0.000s
I20250624 08:02:38.864709 885 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:38.866649 751 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:38.866852 1019 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5e248061ebd6479ca128dd82ead7cc55 | 127.26.229.254:41395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3c4884b8e5b74b6da18da40034a6737c | 127.26.229.194:36983 | HEALTHY | <none> | 1 | 0
5dfd251ca84340fc923e0da615d3252c | 127.26.229.193:38463 | HEALTHY | <none> | 0 | 0
b20f7081883143e38d3713f88507ac8f | 127.26.229.195:45975 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.26.229.193 | experimental | 127.26.229.193:38463
local_ip_for_outbound_sockets | 127.26.229.194 | experimental | 127.26.229.194:36983
local_ip_for_outbound_sockets | 127.26.229.195 | experimental | 127.26.229.195:45975
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb | hidden | 127.26.229.193:38463
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb | hidden | 127.26.229.194:36983
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb | hidden | 127.26.229.195:45975
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet c53ca2985a1c4669a6cc8000a67d8008 of table 'pre_rebuild' is conflicted: 2 replicas' active configs disagree with the leader master's
5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463): RUNNING
b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975): RUNNING
3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983): RUNNING [LEADER]
All reported replicas are:
A = 5dfd251ca84340fc923e0da615d3252c
B = b20f7081883143e38d3713f88507ac8f
C = 3c4884b8e5b74b6da18da40034a6737c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 2 | -1 | Yes
B | A B C | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+--------------------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250624 08:02:38.898238 27543 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250624 08:02:39.094190 751 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:39.097538 1019 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:39.103256 885 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5e248061ebd6479ca128dd82ead7cc55 | 127.26.229.254:41395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3c4884b8e5b74b6da18da40034a6737c | 127.26.229.194:36983 | HEALTHY | <none> | 1 | 0
5dfd251ca84340fc923e0da615d3252c | 127.26.229.193:38463 | HEALTHY | <none> | 0 | 0
b20f7081883143e38d3713f88507ac8f | 127.26.229.195:45975 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.26.229.193 | experimental | 127.26.229.193:38463
local_ip_for_outbound_sockets | 127.26.229.194 | experimental | 127.26.229.194:36983
local_ip_for_outbound_sockets | 127.26.229.195 | experimental | 127.26.229.195:45975
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb | hidden | 127.26.229.193:38463
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb | hidden | 127.26.229.194:36983
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb | hidden | 127.26.229.195:45975
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet c53ca2985a1c4669a6cc8000a67d8008 of table 'pre_rebuild' is conflicted: 2 replicas' active configs disagree with the leader master's
5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463): RUNNING
b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975): RUNNING
3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983): RUNNING [LEADER]
All reported replicas are:
A = 5dfd251ca84340fc923e0da615d3252c
B = b20f7081883143e38d3713f88507ac8f
C = 3c4884b8e5b74b6da18da40034a6737c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 2 | -1 | Yes
B | A B C | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+--------------------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250624 08:02:39.126557 27543 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250624 08:02:39.260349 1039 raft_consensus.cc:1273] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 2 FOLLOWER]: Refusing update from remote peer 3c4884b8e5b74b6da18da40034a6737c: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250624 08:02:39.260973 1159 consensus_queue.cc:1035] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250624 08:02:39.265272 771 raft_consensus.cc:1273] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 2 FOLLOWER]: Refusing update from remote peer 3c4884b8e5b74b6da18da40034a6737c: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250624 08:02:39.265930 1162 consensus_queue.cc:1035] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250624 08:02:39.271564 896 consensus_queue.cc:237] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:39.272594 1039 raft_consensus.cc:1273] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 2 FOLLOWER]: Refusing update from remote peer 3c4884b8e5b74b6da18da40034a6737c: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250624 08:02:39.272994 1162 consensus_queue.cc:1035] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250624 08:02:39.274612 1159 raft_consensus.cc:2953] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 5dfd251ca84340fc923e0da615d3252c (127.26.229.193) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } }
I20250624 08:02:39.275069 1039 raft_consensus.cc:2953] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 5dfd251ca84340fc923e0da615d3252c (127.26.229.193) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } }
I20250624 08:02:39.288398 614 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c53ca2985a1c4669a6cc8000a67d8008 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 08:02:39.289022 630 catalog_manager.cc:5582] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f reported cstate change: config changed from index -1 to 207, VOTER 5dfd251ca84340fc923e0da615d3252c (127.26.229.193) evicted. New cstate: current_term: 2 leader_uuid: "3c4884b8e5b74b6da18da40034a6737c" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } }
I20250624 08:02:39.292551 896 consensus_queue.cc:237] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } }
I20250624 08:02:39.293125 1164 raft_consensus.cc:2953] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER b20f7081883143e38d3713f88507ac8f (127.26.229.195) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } }
I20250624 08:02:39.295223 751 tablet_service.cc:1515] Processing DeleteTablet for tablet c53ca2985a1c4669a6cc8000a67d8008 with delete_type TABLET_DATA_TOMBSTONED (TS 5dfd251ca84340fc923e0da615d3252c not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:53762
I20250624 08:02:39.295707 1173 tablet_replica.cc:331] stopping tablet replica
I20250624 08:02:39.295974 1173 raft_consensus.cc:2241] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250624 08:02:39.296202 1173 raft_consensus.cc:2270] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 08:02:39.297463 1173 ts_tablet_manager.cc:1905] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 08:02:39.385105 614 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c53ca2985a1c4669a6cc8000a67d8008 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 08:02:39.385514 629 catalog_manager.cc:5582] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c reported cstate change: config changed from index 207 to 208, VOTER b20f7081883143e38d3713f88507ac8f (127.26.229.195) evicted. New cstate: current_term: 2 leader_uuid: "3c4884b8e5b74b6da18da40034a6737c" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } health_report { overall_health: HEALTHY } } }
I20250624 08:02:39.389070 1173 ts_tablet_manager.cc:1918] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250624 08:02:39.389185 1173 log.cc:1199] T c53ca2985a1c4669a6cc8000a67d8008 P 5dfd251ca84340fc923e0da615d3252c: Deleting WAL directory at /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/wal/wals/c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:39.389969 617 catalog_manager.cc:4928] TS 5dfd251ca84340fc923e0da615d3252c (127.26.229.193:38463): tablet c53ca2985a1c4669a6cc8000a67d8008 (table pre_rebuild [id=5a92098ee0d34656834ebd9c4ec16a7c]) successfully deleted
I20250624 08:02:39.391949 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet c53ca2985a1c4669a6cc8000a67d8008 with delete_type TABLET_DATA_TOMBSTONED (TS b20f7081883143e38d3713f88507ac8f not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:48390
I20250624 08:02:39.392424 1192 tablet_replica.cc:331] stopping tablet replica
I20250624 08:02:39.392685 1192 raft_consensus.cc:2241] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250624 08:02:39.392948 1192 raft_consensus.cc:2270] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 08:02:39.394279 1192 ts_tablet_manager.cc:1905] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 08:02:39.403612 751 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:39.403887 885 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 08:02:39.403913 1019 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5e248061ebd6479ca128dd82ead7cc55 | 127.26.229.254:41395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3c4884b8e5b74b6da18da40034a6737c | 127.26.229.194:36983 | HEALTHY | <none> | 1 | 0
5dfd251ca84340fc923e0da615d3252c | 127.26.229.193:38463 | HEALTHY | <none> | 0 | 0
b20f7081883143e38d3713f88507ac8f | 127.26.229.195:45975 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.26.229.193 | experimental | 127.26.229.193:38463
local_ip_for_outbound_sockets | 127.26.229.194 | experimental | 127.26.229.194:36983
local_ip_for_outbound_sockets | 127.26.229.195 | experimental | 127.26.229.195:45975
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb | hidden | 127.26.229.193:38463
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb | hidden | 127.26.229.194:36983
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb | hidden | 127.26.229.195:45975
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 08:02:39.487288 27543 log_verifier.cc:126] Checking tablet c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:39.491039 1192 ts_tablet_manager.cc:1918] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250624 08:02:39.491171 1192 log.cc:1199] T c53ca2985a1c4669a6cc8000a67d8008 P b20f7081883143e38d3713f88507ac8f: Deleting WAL directory at /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/wal/wals/c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:39.492146 614 catalog_manager.cc:4928] TS b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975): tablet c53ca2985a1c4669a6cc8000a67d8008 (table pre_rebuild [id=5a92098ee0d34656834ebd9c4ec16a7c]) successfully deleted
I20250624 08:02:39.566259 27543 log_verifier.cc:177] Verified matching terms for 208 ops in tablet c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:39.567476 629 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52744:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250624 08:02:39.567653 629 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52744:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250624 08:02:39.570438 629 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5e248061ebd6479ca128dd82ead7cc55: Sending DeleteTablet for 1 replicas of tablet c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:39.571183 27543 test_util.cc:276] Using random seed: -1064330002
I20250624 08:02:39.571250 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c53ca2985a1c4669a6cc8000a67d8008 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-24 08:02:39 UTC) from {username='slave'} at 127.0.0.1:41694
I20250624 08:02:39.571946 1198 tablet_replica.cc:331] stopping tablet replica
I20250624 08:02:39.572331 1198 raft_consensus.cc:2241] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 LEADER]: Raft consensus shutting down.
I20250624 08:02:39.572597 1198 raft_consensus.cc:2270] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 08:02:39.575892 1198 ts_tablet_manager.cc:1905] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 08:02:39.580394 629 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52830:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 08:02:39.581645 629 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 08:02:39.588847 1198 ts_tablet_manager.cc:1918] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250624 08:02:39.588951 1198 log.cc:1199] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Deleting WAL directory at /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/wal/wals/c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:39.589365 1198 ts_tablet_manager.cc:1939] T c53ca2985a1c4669a6cc8000a67d8008 P 3c4884b8e5b74b6da18da40034a6737c: Deleting consensus metadata
I20250624 08:02:39.591964 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c86343bc1fe547e3b6b7049e50057793 (DEFAULT_TABLE table=post_rebuild [id=2c368026f3054260b1e6468aa11cff01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:39.592015 885 tablet_service.cc:1468] Processing CreateTablet for tablet c86343bc1fe547e3b6b7049e50057793 (DEFAULT_TABLE table=post_rebuild [id=2c368026f3054260b1e6468aa11cff01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:39.592231 751 tablet_service.cc:1468] Processing CreateTablet for tablet c86343bc1fe547e3b6b7049e50057793 (DEFAULT_TABLE table=post_rebuild [id=2c368026f3054260b1e6468aa11cff01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 08:02:39.592548 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86343bc1fe547e3b6b7049e50057793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:39.592598 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86343bc1fe547e3b6b7049e50057793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:39.592701 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86343bc1fe547e3b6b7049e50057793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 08:02:39.596043 1206 tablet_bootstrap.cc:492] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c: Bootstrap starting.
I20250624 08:02:39.596235 1205 tablet_bootstrap.cc:492] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f: Bootstrap starting.
I20250624 08:02:39.596823 1206 tablet_bootstrap.cc:654] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:39.597640 1205 tablet_bootstrap.cc:654] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:39.597728 1206 tablet_bootstrap.cc:492] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c: No bootstrap required, opened a new log
I20250624 08:02:39.597824 1206 ts_tablet_manager.cc:1397] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:39.598233 1206 raft_consensus.cc:357] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.598471 1204 tablet_bootstrap.cc:492] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c: Bootstrap starting.
I20250624 08:02:39.598753 1206 raft_consensus.cc:383] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:39.598887 1206 raft_consensus.cc:738] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4884b8e5b74b6da18da40034a6737c, State: Initialized, Role: FOLLOWER
I20250624 08:02:39.599088 1206 consensus_queue.cc:260] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.599445 1206 ts_tablet_manager.cc:1428] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250624 08:02:39.599699 1204 tablet_bootstrap.cc:654] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c: Neither blocks nor log segments found. Creating new log.
I20250624 08:02:39.601023 1204 tablet_bootstrap.cc:492] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c: No bootstrap required, opened a new log
I20250624 08:02:39.601135 1204 ts_tablet_manager.cc:1397] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250624 08:02:39.601814 1204 raft_consensus.cc:357] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.601950 1204 raft_consensus.cc:383] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:39.601998 1204 raft_consensus.cc:738] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfd251ca84340fc923e0da615d3252c, State: Initialized, Role: FOLLOWER
I20250624 08:02:39.602157 1204 consensus_queue.cc:260] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.603106 1204 ts_tablet_manager.cc:1428] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250624 08:02:39.603641 1205 tablet_bootstrap.cc:492] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f: No bootstrap required, opened a new log
I20250624 08:02:39.603727 1205 ts_tablet_manager.cc:1397] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250624 08:02:39.604189 1205 raft_consensus.cc:357] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.604303 1205 raft_consensus.cc:383] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 08:02:39.604350 1205 raft_consensus.cc:738] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20f7081883143e38d3713f88507ac8f, State: Initialized, Role: FOLLOWER
I20250624 08:02:39.604470 1205 consensus_queue.cc:260] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.604682 1205 ts_tablet_manager.cc:1428] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 08:02:39.621471 1166 raft_consensus.cc:491] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 08:02:39.621634 1166 raft_consensus.cc:513] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.623006 1166 leader_election.cc:290] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975), 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983)
I20250624 08:02:39.627028 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86343bc1fe547e3b6b7049e50057793" candidate_uuid: "5dfd251ca84340fc923e0da615d3252c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4884b8e5b74b6da18da40034a6737c" is_pre_election: true
I20250624 08:02:39.627261 896 raft_consensus.cc:2466] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dfd251ca84340fc923e0da615d3252c in term 0.
I20250624 08:02:39.627507 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86343bc1fe547e3b6b7049e50057793" candidate_uuid: "5dfd251ca84340fc923e0da615d3252c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20f7081883143e38d3713f88507ac8f" is_pre_election: true
I20250624 08:02:39.627651 1039 raft_consensus.cc:2466] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dfd251ca84340fc923e0da615d3252c in term 0.
I20250624 08:02:39.627655 703 leader_election.cc:304] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4884b8e5b74b6da18da40034a6737c, 5dfd251ca84340fc923e0da615d3252c; no voters:
I20250624 08:02:39.627853 1166 raft_consensus.cc:2802] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 08:02:39.627907 1166 raft_consensus.cc:491] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 08:02:39.627943 1166 raft_consensus.cc:3058] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:39.634025 614 catalog_manager.cc:4928] TS 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983): tablet c53ca2985a1c4669a6cc8000a67d8008 (table pre_rebuild [id=5a92098ee0d34656834ebd9c4ec16a7c]) successfully deleted
I20250624 08:02:39.689160 1166 raft_consensus.cc:513] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.689740 1166 leader_election.cc:290] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [CANDIDATE]: Term 1 election: Requested vote from peers b20f7081883143e38d3713f88507ac8f (127.26.229.195:45975), 3c4884b8e5b74b6da18da40034a6737c (127.26.229.194:36983)
I20250624 08:02:39.690210 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86343bc1fe547e3b6b7049e50057793" candidate_uuid: "5dfd251ca84340fc923e0da615d3252c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20f7081883143e38d3713f88507ac8f"
I20250624 08:02:39.690387 1039 raft_consensus.cc:3058] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 0 FOLLOWER]: Advancing to term 1
I20250624 08:02:39.690408 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86343bc1fe547e3b6b7049e50057793" candidate_uuid: "5dfd251ca84340fc923e0da615d3252c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4884b8e5b74b6da18da40034a6737c"
I20250624 08:02:39.690570 896 raft_consensus.cc:3058] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 0 FOLLOWER]: Advancing to term 1
W20250624 08:02:39.691574 1086 tablet.cc:2378] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:39.691674 1039 raft_consensus.cc:2466] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dfd251ca84340fc923e0da615d3252c in term 1.
I20250624 08:02:39.691876 896 raft_consensus.cc:2466] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dfd251ca84340fc923e0da615d3252c in term 1.
I20250624 08:02:39.692060 703 leader_election.cc:304] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfd251ca84340fc923e0da615d3252c, b20f7081883143e38d3713f88507ac8f; no voters:
I20250624 08:02:39.692389 1166 raft_consensus.cc:2802] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 1 FOLLOWER]: Leader election won for term 1
I20250624 08:02:39.692618 1166 raft_consensus.cc:695] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [term 1 LEADER]: Becoming Leader. State: Replica: 5dfd251ca84340fc923e0da615d3252c, State: Running, Role: LEADER
I20250624 08:02:39.692934 1166 consensus_queue.cc:237] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } }
I20250624 08:02:39.696087 629 catalog_manager.cc:5582] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dfd251ca84340fc923e0da615d3252c (127.26.229.193). New cstate: current_term: 1 leader_uuid: "5dfd251ca84340fc923e0da615d3252c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dfd251ca84340fc923e0da615d3252c" member_type: VOTER last_known_addr { host: "127.26.229.193" port: 38463 } health_report { overall_health: HEALTHY } } }
W20250624 08:02:39.711393 952 tablet.cc:2378] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 08:02:39.735865 1039 raft_consensus.cc:1273] T c86343bc1fe547e3b6b7049e50057793 P b20f7081883143e38d3713f88507ac8f [term 1 FOLLOWER]: Refusing update from remote peer 5dfd251ca84340fc923e0da615d3252c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:02:39.736289 1166 consensus_queue.cc:1035] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20f7081883143e38d3713f88507ac8f" member_type: VOTER last_known_addr { host: "127.26.229.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:39.739343 896 raft_consensus.cc:1273] T c86343bc1fe547e3b6b7049e50057793 P 3c4884b8e5b74b6da18da40034a6737c [term 1 FOLLOWER]: Refusing update from remote peer 5dfd251ca84340fc923e0da615d3252c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 08:02:39.740412 1166 consensus_queue.cc:1035] T c86343bc1fe547e3b6b7049e50057793 P 5dfd251ca84340fc923e0da615d3252c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4884b8e5b74b6da18da40034a6737c" member_type: VOTER last_known_addr { host: "127.26.229.194" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 08:02:39.740554 1169 mvcc.cc:204] Tried to move back new op lower bound from 7171080846275948544 to 7171080846105419776. Current Snapshot: MvccSnapshot[applied={T|T < 7171080846272163840}]
I20250624 08:02:39.742271 1165 mvcc.cc:204] Tried to move back new op lower bound from 7171080846275948544 to 7171080846105419776. Current Snapshot: MvccSnapshot[applied={T|T < 7171080846272163840}]
I20250624 08:02:39.748040 1163 mvcc.cc:204] Tried to move back new op lower bound from 7171080846272163840 to 7171080846105419776. Current Snapshot: MvccSnapshot[applied={T|T < 7171080846272163840}]
I20250624 08:02:40.534662 885 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:40.535434 1019 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 08:02:40.536041 751 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5e248061ebd6479ca128dd82ead7cc55 | 127.26.229.254:41395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
3c4884b8e5b74b6da18da40034a6737c | 127.26.229.194:36983 | HEALTHY | <none> | 0 | 0
5dfd251ca84340fc923e0da615d3252c | 127.26.229.193:38463 | HEALTHY | <none> | 1 | 0
b20f7081883143e38d3713f88507ac8f | 127.26.229.195:45975 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.26.229.193 | experimental | 127.26.229.193:38463
local_ip_for_outbound_sockets | 127.26.229.194 | experimental | 127.26.229.194:36983
local_ip_for_outbound_sockets | 127.26.229.195 | experimental | 127.26.229.195:45975
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-0/data/info.pb | hidden | 127.26.229.193:38463
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-1/data/info.pb | hidden | 127.26.229.194:36983
server_dump_info_path | /tmp/dist-test-task7veRrf/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750752133635324-27543-0/minicluster-data/ts-2/data/info.pb | hidden | 127.26.229.195:45975
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.26.229.212:32991 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 08:02:40.557914 27543 log_verifier.cc:126] Checking tablet c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:40.558044 27543 log_verifier.cc:177] Verified matching terms for 0 ops in tablet c53ca2985a1c4669a6cc8000a67d8008
I20250624 08:02:40.558089 27543 log_verifier.cc:126] Checking tablet c86343bc1fe547e3b6b7049e50057793
I20250624 08:02:40.649644 27543 log_verifier.cc:177] Verified matching terms for 206 ops in tablet c86343bc1fe547e3b6b7049e50057793
I20250624 08:02:40.652751 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 668
I20250624 08:02:40.659220 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 821
I20250624 08:02:40.665696 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 954
I20250624 08:02:40.671526 27543 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7veRrf/build/debug/bin/kudu with pid 599
2025-06-24T08:02:40Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (6479 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (6479 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (27052 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST