[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:24 dist-test-slave-3h5h krb5kdc[29393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:24 dist-test-slave-3h5h krb5kdc[29393](info): set up 2 sockets
Aug 08 02:17:24 dist-test-slave-3h5h krb5kdc[29393](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 02:17:25.452546 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:25 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:25.589422 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36437
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:25.685122 29408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:25.685400 29408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:25.685448 29408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:25.686794 29408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:25.686867 29408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:25.686892 29408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:25.686911 29408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:25.686944 29408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:25.688663 29408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:36437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:25.688999 29408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:25.689249 29408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:25.694630 29415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:25.694684 29418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:25.694687 29416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:25.695142 29408 server_base.cc:1034] running on GCE node
I20260808 02:17:25.695549 29408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:25.695962 29408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:25.697163 29408 hybrid_clock.cc:648] HybridClock initialized: now 1786155445697133 us; error 54 us; skew 500 ppm
Aug 08 02:17:25 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:25.706480 29408 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:25.707202 29408 webserver.cc:460] Webserver started at http://127.28.177.190:39387/ using document root <none> and password file <none>
I20260808 02:17:25.707463 29408 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:25.707540 29408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:25.707726 29408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:25.709033 29408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "7af480fd85424197b1071494209a595e"
format_stamp: "Formatted at 2026-08-08 02:17:25 on dist-test-slave-3h5h"
server_key: "1c8b122e02ad58b7f5be759ca49e8f3d"
server_key_iv: "6c09c3bfbe63bbf39fe21b82285a220e"
server_key_version: "encryptionkey@0"
I20260808 02:17:25.709431 29408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "7af480fd85424197b1071494209a595e"
format_stamp: "Formatted at 2026-08-08 02:17:25 on dist-test-slave-3h5h"
server_key: "1c8b122e02ad58b7f5be759ca49e8f3d"
server_key_iv: "6c09c3bfbe63bbf39fe21b82285a220e"
server_key_version: "encryptionkey@0"
I20260808 02:17:25.711093 29408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:17:25.712114 29427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:25.712383 29408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:25.712548 29408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "7af480fd85424197b1071494209a595e"
format_stamp: "Formatted at 2026-08-08 02:17:25 on dist-test-slave-3h5h"
server_key: "1c8b122e02ad58b7f5be759ca49e8f3d"
server_key_iv: "6c09c3bfbe63bbf39fe21b82285a220e"
server_key_version: "encryptionkey@0"
I20260808 02:17:25.712636 29408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:25.736245 29408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:25.738790 29408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:25.738972 29408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:25.743666 29408 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:36437
I20260808 02:17:25.743705 29502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:36437 every 8 connection(s)
I20260808 02:17:25.744158 29408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:25.745261 29505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:25.746320 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29408
I20260808 02:17:25.746443 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:25.747152 29382 external_mini_cluster.cc:1428] Setting key 36a138042887729ddf945fb68eb4a517
I20260808 02:17:25.747996 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e: Bootstrap starting.
I20260808 02:17:25.748780 29505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:25.749178 29505 log.cc:824] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:25.750020 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e: No bootstrap required, opened a new log
I20260808 02:17:25.752030 29505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af480fd85424197b1071494209a595e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36437 } }
I20260808 02:17:25.752218 29505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:25.752314 29505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af480fd85424197b1071494209a595e, State: Initialized, Role: FOLLOWER
I20260808 02:17:25.752460 29505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af480fd85424197b1071494209a595e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36437 } }
I20260808 02:17:25.752549 29505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:25.752581 29505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:25.752635 29505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:25.753894 29505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af480fd85424197b1071494209a595e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36437 } }
I20260808 02:17:25.754025 29505 leader_election.cc:302] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af480fd85424197b1071494209a595e; no voters:
I20260808 02:17:25.754217 29505 leader_election.cc:288] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:25.754289 29516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:25.754460 29516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [term 1 LEADER]: Becoming Leader. State: Replica: 7af480fd85424197b1071494209a595e, State: Running, Role: LEADER
I20260808 02:17:25.754494 29505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:25.754577 29516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af480fd85424197b1071494209a595e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36437 } }
I20260808 02:17:25.754911 29516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af480fd85424197b1071494209a595e. Latest consensus state: current_term: 1 leader_uuid: "7af480fd85424197b1071494209a595e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af480fd85424197b1071494209a595e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36437 } } }
I20260808 02:17:25.755002 29516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:25.754923 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af480fd85424197b1071494209a595e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af480fd85424197b1071494209a595e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36437 } } }
I20260808 02:17:25.755370 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:25.756108 29524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:25.756620 29524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:25.758667 29524 catalog_manager.cc:1347] Generated new cluster ID: b54dfc105fc1409a9759a4d1f1609fc2
I20260808 02:17:25.758738 29524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:17:25 dist-test-slave-3h5h krb5kdc[29393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:25.767462 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:25.748179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.28.177.190:36437)
0808 02:17:25.748370 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:25.748380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:25.748629 (+ 249us) server_negotiation.cc:407] Connection header received
0808 02:17:25.752013 (+ 3384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:25.752021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:25.752101 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:25.752265 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:25.754793 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:25.756618 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:25.758225 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:25.758703 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:25.758835 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:25.762953 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:25.762957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:25.762957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:25.762968 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:25.765422 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:25.765888 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:25.765891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:25.765891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:25.765988 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:25.766276 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:25.766277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:25.766277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:25.766539 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:25.766724 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:25.767008 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:17:25.767210 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260808 02:17:25.772270 29524 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:25.772995 29524 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:25.782117 29524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af480fd85424197b1071494209a595e: Generated new TSK 0
I20260808 02:17:25.782366 29524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:25.820379 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36437
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:25.922533 29546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:25.922762 29546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:25.922786 29546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:25.923977 29546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:25.924026 29546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:25.924053 29546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:25.925648 29546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:25.925904 29546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:25.926177 29546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:25.931649 29557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:25.931672 29560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:25.931660 29558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:25.931699 29546 server_base.cc:1034] running on GCE node
I20260808 02:17:25.932127 29546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:25.932371 29546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:25.933533 29546 hybrid_clock.cc:648] HybridClock initialized: now 1786155445933507 us; error 42 us; skew 500 ppm
Aug 08 02:17:25 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:25.936304 29546 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:25.936753 29546 webserver.cc:460] Webserver started at http://127.28.177.129:37145/ using document root <none> and password file <none>
I20260808 02:17:25.937026 29546 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:25.937073 29546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:25.937204 29546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:25.938292 29546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "35cfcc1f521a4cfa97516fb61479537a"
format_stamp: "Formatted at 2026-08-08 02:17:25 on dist-test-slave-3h5h"
server_key: "b7e92d9d2a8cee5901b2f200381739e3"
server_key_iv: "fb27296747f700a19bb465f2675e7117"
server_key_version: "encryptionkey@0"
I20260808 02:17:25.938665 29546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "35cfcc1f521a4cfa97516fb61479537a"
format_stamp: "Formatted at 2026-08-08 02:17:25 on dist-test-slave-3h5h"
server_key: "b7e92d9d2a8cee5901b2f200381739e3"
server_key_iv: "fb27296747f700a19bb465f2675e7117"
server_key_version: "encryptionkey@0"
I20260808 02:17:25.940171 29546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:25.940965 29569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:25.941195 29546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:25.941291 29546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "35cfcc1f521a4cfa97516fb61479537a"
format_stamp: "Formatted at 2026-08-08 02:17:25 on dist-test-slave-3h5h"
server_key: "b7e92d9d2a8cee5901b2f200381739e3"
server_key_iv: "fb27296747f700a19bb465f2675e7117"
server_key_version: "encryptionkey@0"
I20260808 02:17:25.941375 29546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:25.951435 29546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:25.954222 29546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:25.954499 29546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:25.954850 29546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:25.955224 29546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:25.955298 29546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:25.955341 29546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:25.955366 29546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:25.962021 29546 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:42199
I20260808 02:17:25.962085 29727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:42199 every 8 connection(s)
I20260808 02:17:25.962445 29546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:25.967643 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29546
I20260808 02:17:25.967747 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:25.967911 29382 external_mini_cluster.cc:1428] Setting key 9dc307b700a6c4732b98d82a123d13c9
Aug 08 02:17:25 dist-test-slave-3h5h krb5kdc[29393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:25.972786 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:25.963198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:39249 (local address 127.28.177.190:36437)
0808 02:17:25.963265 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:25.963268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:25.963494 (+ 226us) server_negotiation.cc:407] Connection header received
0808 02:17:25.963987 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:25.963988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:25.964001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:25.964057 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:25.965176 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:25.965687 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:25.966786 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:25.966986 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:25.967043 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:25.969642 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:25.969643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:25.969644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:25.969653 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:25.971133 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:25.971606 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:25.971608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:25.971608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:25.971709 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:25.972049 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:25.972051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:25.972051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:25.972224 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:25.972293 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:25.972642 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:17:25.972719 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:17:25.973160 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:25.963178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36437 (local address 127.28.177.129:39249)
0808 02:17:25.963371 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:25.963408 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:25.963898 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:25.964111 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:25.964114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:25.964759 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:25.965040 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:25.965045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:25.965778 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:25.965779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:25.966695 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:25.966697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:25.967113 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:25.967114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:25.967215 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:25.967718 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:25.967731 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:25.969528 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:25.971240 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:25.971249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:25.971251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:25.971491 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:25.971786 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:25.971787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:25.971788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:25.971942 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:25.972304 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:25.972311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:25.972488 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 02:17:25.972615 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 02:17:25.972818 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
I20260808 02:17:25.973757 29730 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36437
I20260808 02:17:25.973888 29730 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:25.974205 29730 heartbeater.cc:511] Master 127.28.177.190:36437 requested a full tablet report, sending...
I20260808 02:17:25.974965 29452 ts_manager.cc:194] Registered new tserver with Master: 35cfcc1f521a4cfa97516fb61479537a (127.28.177.129:42199)
I20260808 02:17:25.976001 29452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:39249
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:26.000706 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36437
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:26.089828 29736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:26.090287 29736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:26.090339 29736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:26.091789 29736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:26.091871 29736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:26.091938 29736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:26.093771 29736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:26.094105 29736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:26.094403 29736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:26.100332 29743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:26.100361 29744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:26.100332 29747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:26.100550 29736 server_base.cc:1034] running on GCE node
I20260808 02:17:26.100764 29736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:26.101048 29736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:26.102260 29736 hybrid_clock.cc:648] HybridClock initialized: now 1786155446102250 us; error 42 us; skew 500 ppm
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:26.105003 29736 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:26.105502 29736 webserver.cc:460] Webserver started at http://127.28.177.130:37993/ using document root <none> and password file <none>
I20260808 02:17:26.105762 29736 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:26.105834 29736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:26.105983 29736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:26.107031 29736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "536c3bfde6af4840844f808299f4448c"
format_stamp: "Formatted at 2026-08-08 02:17:26 on dist-test-slave-3h5h"
server_key: "9ab906b843ac4226730c76a857a64ae3"
server_key_iv: "ec6f15f3e5f2d2b5bda5bd8f9d36594c"
server_key_version: "encryptionkey@0"
I20260808 02:17:26.107434 29736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "536c3bfde6af4840844f808299f4448c"
format_stamp: "Formatted at 2026-08-08 02:17:26 on dist-test-slave-3h5h"
server_key: "9ab906b843ac4226730c76a857a64ae3"
server_key_iv: "ec6f15f3e5f2d2b5bda5bd8f9d36594c"
server_key_version: "encryptionkey@0"
I20260808 02:17:26.108824 29736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:26.109766 29760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:26.109951 29736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:26.110116 29736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "536c3bfde6af4840844f808299f4448c"
format_stamp: "Formatted at 2026-08-08 02:17:26 on dist-test-slave-3h5h"
server_key: "9ab906b843ac4226730c76a857a64ae3"
server_key_iv: "ec6f15f3e5f2d2b5bda5bd8f9d36594c"
server_key_version: "encryptionkey@0"
I20260808 02:17:26.110204 29736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:26.119794 29736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:26.122177 29736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:26.122347 29736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:26.122587 29736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:26.122907 29736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:26.122963 29736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:26.123005 29736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:26.123034 29736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:26.129522 29736 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:41823
I20260808 02:17:26.129582 29926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:41823 every 8 connection(s)
I20260808 02:17:26.129904 29736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:26.137075 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29736
I20260808 02:17:26.137230 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:26.137447 29382 external_mini_cluster.cc:1428] Setting key b0932c926986680c59265c827d8c60c9
I20260808 02:17:26.140050 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:26.130688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:57423 (local address 127.28.177.190:36437)
0808 02:17:26.130743 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:26.130745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:26.131015 (+ 270us) server_negotiation.cc:407] Connection header received
0808 02:17:26.131541 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:26.131543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:26.131553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:26.131616 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:26.132621 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.133146 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.134275 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.134458 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.134511 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:26.136804 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:26.136806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:26.136806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:26.136813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:26.138626 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.139055 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.139057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.139057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.139126 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.139383 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.139384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.139385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.139507 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:26.139586 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:26.139862 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 02:17:26.139959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:26.140251 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:26.130664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36437 (local address 127.28.177.130:57423)
0808 02:17:26.130851 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:26.130899 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:26.131446 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:26.131674 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:26.131677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:26.132319 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:26.132523 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:26.132538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.133233 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.133234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:26.134190 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:26.134193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.134567 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.134568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:26.134679 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:26.135166 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:26.135175 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:26.136716 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:26.138712 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:26.138720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:26.138723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:26.138957 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:26.139196 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:26.139197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:26.139198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:26.139311 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:26.139580 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:26.139587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:26.139729 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 02:17:26.139840 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:17:26.140000 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
I20260808 02:17:26.140992 29927 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36437
I20260808 02:17:26.141105 29927 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:26.141290 29927 heartbeater.cc:511] Master 127.28.177.190:36437 requested a full tablet report, sending...
I20260808 02:17:26.141776 29452 ts_manager.cc:194] Registered new tserver with Master: 536c3bfde6af4840844f808299f4448c (127.28.177.130:41823)
I20260808 02:17:26.142155 29452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:57423
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:26.169090 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36437
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:26.254942 29934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:26.255206 29934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:26.255277 29934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:26.256611 29934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:26.256659 29934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:26.256686 29934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:26.258188 29934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36437
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:26.258381 29934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:26.258551 29934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:26.264339 29943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:26.264374 29934 server_base.cc:1034] running on GCE node
W20260808 02:17:26.264478 29942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:26.264590 29946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:26.265022 29934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:26.265285 29934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:26.266461 29934 hybrid_clock.cc:648] HybridClock initialized: now 1786155446266451 us; error 47 us; skew 500 ppm
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:26.269160 29934 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:26.269619 29934 webserver.cc:460] Webserver started at http://127.28.177.131:41571/ using document root <none> and password file <none>
I20260808 02:17:26.269853 29934 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:26.269923 29934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:26.270085 29934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:26.271157 29934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "7753e3c90e004c8e8bd9ccd5868a365d"
format_stamp: "Formatted at 2026-08-08 02:17:26 on dist-test-slave-3h5h"
server_key: "c1b122abde381d60377cf89b300f5fac"
server_key_iv: "fb73500b12356a700ee7b0f5044033a3"
server_key_version: "encryptionkey@0"
I20260808 02:17:26.271567 29934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "7753e3c90e004c8e8bd9ccd5868a365d"
format_stamp: "Formatted at 2026-08-08 02:17:26 on dist-test-slave-3h5h"
server_key: "c1b122abde381d60377cf89b300f5fac"
server_key_iv: "fb73500b12356a700ee7b0f5044033a3"
server_key_version: "encryptionkey@0"
I20260808 02:17:26.272969 29934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:17:26.273756 29954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:26.273928 29934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:26.274039 29934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "7753e3c90e004c8e8bd9ccd5868a365d"
format_stamp: "Formatted at 2026-08-08 02:17:26 on dist-test-slave-3h5h"
server_key: "c1b122abde381d60377cf89b300f5fac"
server_key_iv: "fb73500b12356a700ee7b0f5044033a3"
server_key_version: "encryptionkey@0"
I20260808 02:17:26.274118 29934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:26.304248 29934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:26.306824 29934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:26.307000 29934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:26.307258 29934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:26.307603 29934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:26.307665 29934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:26.307711 29934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:26.307736 29934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:26.314229 29934 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:37365
I20260808 02:17:26.314302 30115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:37365 every 8 connection(s)
I20260808 02:17:26.314646 29934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:17:26.315531 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29934
I20260808 02:17:26.315639 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:26.315811 29382 external_mini_cluster.cc:1428] Setting key eb9b0881f412374a1d56d2b11a257586
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:26.324136 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:26.315510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:53129 (local address 127.28.177.190:36437)
0808 02:17:26.315557 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:26.315559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:26.315867 (+ 308us) server_negotiation.cc:407] Connection header received
0808 02:17:26.316537 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:26.316538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:26.316551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:26.316636 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:26.317791 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.318300 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.319272 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.319450 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.319501 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:26.321705 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:26.321706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:26.321707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:26.321714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:26.322979 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.323349 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.323350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.323350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.323400 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.323622 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.323622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.323623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.323726 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:26.323778 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:26.324006 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 02:17:26.324069 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:17:26.324419 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:26.315489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36437 (local address 127.28.177.131:53129)
0808 02:17:26.315708 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:26.315761 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:26.316443 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:26.316742 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:26.316747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:26.317496 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:26.317697 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:26.317701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.318389 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.318390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:26.319191 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:26.319193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.319522 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.319522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:26.319627 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:26.320175 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:26.320184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:26.321606 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:26.323062 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:26.323068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:26.323069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:26.323282 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:26.323457 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:26.323458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:26.323458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:26.323565 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:26.323781 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:26.323785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:26.323895 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 02:17:26.323978 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:17:26.324135 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260808 02:17:26.324977 30120 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36437
I20260808 02:17:26.325116 30120 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:26.325269 30120 heartbeater.cc:511] Master 127.28.177.190:36437 requested a full tablet report, sending...
I20260808 02:17:26.325691 29452 ts_manager.cc:194] Registered new tserver with Master: 7753e3c90e004c8e8bd9ccd5868a365d (127.28.177.131:37365)
I20260808 02:17:26.326297 29452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:53129
I20260808 02:17:26.328121 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:26.353359 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:26.345231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.28.177.129:42199)
0808 02:17:26.345478 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:26.345483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:26.345494 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:26.345523 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:26.345525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:26.345540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:26.345610 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:26.346210 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.346943 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.347693 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.348021 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.348088 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:26.350555 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:26.350571 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:26.350571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:26.350579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:26.352145 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.352477 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.352480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.352481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.352536 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.352706 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.352706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.352706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.353029 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:26.353174 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:26.353211 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:17:26.353278 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":114,"threads_started":1}
W20260808 02:17:26.353849 29605 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49550
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[29393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:26.378769 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:26.371515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.28.177.129:42199)
0808 02:17:26.371627 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:26.371629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:26.371724 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:17:26.371776 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:26.371777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:26.371788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:26.371842 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:26.372401 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.373016 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.373715 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:26.373907 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:26.373959 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:26.376172 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:26.376174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:26.376174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:26.376181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:26.377801 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.378120 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.378121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.378121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.378175 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:26.378340 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:26.378340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:26.378341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:26.378430 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:26.378514 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:26.378595 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 02:17:26.378661 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:17:26.379449 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29546
I20260808 02:17:26.391039 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29736
I20260808 02:17:26.414494 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29934
I20260808 02:17:26.427989 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29408
2026-08-08T02:17:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2322 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[30143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[30143](info): set up 2 sockets
Aug 08 02:17:26 dist-test-slave-3h5h krb5kdc[30143](info): commencing operation
krb5kdc: starting...
W20260808 02:17:27.742337 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:27 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:27.877081 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36365
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:27.967120 30164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:27.967382 30164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:27.967427 30164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:27.968664 30164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:27.968734 30164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:27.968758 30164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:27.968793 30164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:27.968822 30164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:27.970458 30164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:27.970757 30164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:27.971004 30164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:27.976605 30170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:27.976583 30173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:27.976620 30164 server_base.cc:1034] running on GCE node
W20260808 02:17:27.976583 30171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:27.977262 30164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:27.977594 30164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:27.978780 30164 hybrid_clock.cc:648] HybridClock initialized: now 1786155447978757 us; error 42 us; skew 500 ppm
Aug 08 02:17:27 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155447, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:27.981568 30164 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:27.982080 30164 webserver.cc:460] Webserver started at http://127.28.177.190:39553/ using document root <none> and password file <none>
I20260808 02:17:27.982282 30164 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:27.982373 30164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:27.982542 30164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:27.983691 30164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "9abef6501db64f0aadcaf2cf73780856"
format_stamp: "Formatted at 2026-08-08 02:17:27 on dist-test-slave-3h5h"
server_key: "b3dbd76ca94cf7a2dc15b1cc38ca2542"
server_key_iv: "f22002992f8be2885a4d23a398d80bc6"
server_key_version: "encryptionkey@0"
I20260808 02:17:27.984071 30164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "9abef6501db64f0aadcaf2cf73780856"
format_stamp: "Formatted at 2026-08-08 02:17:27 on dist-test-slave-3h5h"
server_key: "b3dbd76ca94cf7a2dc15b1cc38ca2542"
server_key_iv: "f22002992f8be2885a4d23a398d80bc6"
server_key_version: "encryptionkey@0"
I20260808 02:17:27.985682 30164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 02:17:27.986632 30181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:27.986869 30164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:27.987082 30164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "9abef6501db64f0aadcaf2cf73780856"
format_stamp: "Formatted at 2026-08-08 02:17:27 on dist-test-slave-3h5h"
server_key: "b3dbd76ca94cf7a2dc15b1cc38ca2542"
server_key_iv: "f22002992f8be2885a4d23a398d80bc6"
server_key_version: "encryptionkey@0"
I20260808 02:17:27.987174 30164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:28.000715 30164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:28.002990 30164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:28.003229 30164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:28.007581 30164 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:36365
I20260808 02:17:28.007622 30252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:36365 every 8 connection(s)
I20260808 02:17:28.007998 30164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:28.009032 30253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:28.011639 30253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856: Bootstrap starting.
I20260808 02:17:28.012321 30253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:28.012681 30253 log.cc:824] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:28.013450 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 30164
I20260808 02:17:28.013525 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:28.013716 29382 external_mini_cluster.cc:1428] Setting key 99f1fd468366dd88f63f9be612e00f68
I20260808 02:17:28.013713 30253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856: No bootstrap required, opened a new log
I20260808 02:17:28.015947 30253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abef6501db64f0aadcaf2cf73780856" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36365 } }
I20260808 02:17:28.016127 30253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:28.016181 30253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abef6501db64f0aadcaf2cf73780856, State: Initialized, Role: FOLLOWER
I20260808 02:17:28.016331 30253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abef6501db64f0aadcaf2cf73780856" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36365 } }
I20260808 02:17:28.016440 30253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:28.016479 30253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:28.016542 30253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:28.017721 30253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abef6501db64f0aadcaf2cf73780856" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36365 } }
I20260808 02:17:28.017823 30253 leader_election.cc:302] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abef6501db64f0aadcaf2cf73780856; no voters:
I20260808 02:17:28.018025 30253 leader_election.cc:288] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:28.018077 30262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:28.018214 30262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [term 1 LEADER]: Becoming Leader. State: Replica: 9abef6501db64f0aadcaf2cf73780856, State: Running, Role: LEADER
I20260808 02:17:28.018301 30262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abef6501db64f0aadcaf2cf73780856" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36365 } }
I20260808 02:17:28.018344 30253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:28.018640 30265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abef6501db64f0aadcaf2cf73780856. Latest consensus state: current_term: 1 leader_uuid: "9abef6501db64f0aadcaf2cf73780856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abef6501db64f0aadcaf2cf73780856" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36365 } } }
I20260808 02:17:28.018715 30265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:28.018921 30267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:28.019594 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9abef6501db64f0aadcaf2cf73780856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abef6501db64f0aadcaf2cf73780856" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36365 } } }
I20260808 02:17:28.019675 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:28.020141 30267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:28.021853 30267 catalog_manager.cc:1347] Generated new cluster ID: 3c24fec8535f4a578c3df7e6d59acda3
I20260808 02:17:28.021903 30267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:28.027436 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.014499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.28.177.190:36365)
0808 02:17:28.014699 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.014708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.014734 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:17:28.015927 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.015935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.016085 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.016283 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.017181 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.018390 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.019634 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.020401 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.020523 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.023692 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:28.023697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:28.023697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:28.023710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:28.025708 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.026129 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.026137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.026137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.026256 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.026491 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.026492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.026493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.026779 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:28.026991 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.027064 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.027225 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260808 02:17:28.038743 30267 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:28.039477 30267 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:28.045006 30267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abef6501db64f0aadcaf2cf73780856: Generated new TSK 0
I20260808 02:17:28.045202 30267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:28.076951 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36365
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:28.169173 30290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:28.169443 30290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:28.169490 30290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:28.170809 30290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:28.170881 30290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:28.170928 30290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:28.172672 30290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:28.172957 30290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:28.173207 30290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:28.178383 30298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:28.178426 30295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:28.178427 30300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:28.178666 30290 server_base.cc:1034] running on GCE node
I20260808 02:17:28.178897 30290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:28.179159 30290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:28.180310 30290 hybrid_clock.cc:648] HybridClock initialized: now 1786155448180295 us; error 32 us; skew 500 ppm
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:28.183274 30290 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:28.183748 30290 webserver.cc:460] Webserver started at http://127.28.177.129:45147/ using document root <none> and password file <none>
I20260808 02:17:28.183986 30290 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:28.184031 30290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:28.184211 30290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:28.185369 30290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "3fedff30870949fd9b5f03804fd5f5f6"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "7925b14290b0d79c58c6aac8ce6b383b"
server_key_iv: "f9230a7591e6ea7b8e12e7aa6751453f"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.185750 30290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "3fedff30870949fd9b5f03804fd5f5f6"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "7925b14290b0d79c58c6aac8ce6b383b"
server_key_iv: "f9230a7591e6ea7b8e12e7aa6751453f"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.187323 30290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 02:17:28.188237 30311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.188517 30290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:28.188638 30290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "3fedff30870949fd9b5f03804fd5f5f6"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "7925b14290b0d79c58c6aac8ce6b383b"
server_key_iv: "f9230a7591e6ea7b8e12e7aa6751453f"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.188717 30290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:28.199939 30290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:28.202497 30290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:28.202701 30290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:28.202991 30290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:28.203399 30290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:28.203471 30290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.203524 30290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:28.203583 30290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.210618 30290 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:32897
I20260808 02:17:28.210700 30470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:32897 every 8 connection(s)
I20260808 02:17:28.211091 30290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:28.213359 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 30290
I20260808 02:17:28.213456 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:28.213642 29382 external_mini_cluster.cc:1428] Setting key 530f9b68ba9afdb672ec80e2e4411211
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:28.223129 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.211969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53859 (local address 127.28.177.190:36365)
0808 02:17:28.212042 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.212045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.212307 (+ 262us) server_negotiation.cc:407] Connection header received
0808 02:17:28.213611 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.213613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.213629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.213736 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.214570 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.215100 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.216557 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.216788 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.216915 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.219503 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:28.219505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:28.219505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:28.219516 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:28.221367 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.221823 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.221825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.221826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.221897 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.222316 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.222318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.222318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.222503 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:28.222596 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.222905 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.223011 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:28.223261 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.211964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36365 (local address 127.28.177.129:53859)
0808 02:17:28.212177 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.212215 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.213495 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.213822 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.213827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.214185 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:28.214448 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.214454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.215226 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.215227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.216463 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.216466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.216927 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.216928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.217051 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.217608 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:28.217617 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:28.219405 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:28.221479 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:28.221488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:28.221490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:28.221741 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:28.222087 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:28.222088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:28.222088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:28.222217 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:28.222599 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:28.222607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:28.222754 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.222852 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.222988 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260808 02:17:28.223832 30474 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36365
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:17:28.223939 30474 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:28.224171 30474 heartbeater.cc:511] Master 127.28.177.190:36365 requested a full tablet report, sending...
I20260808 02:17:28.224726 30200 ts_manager.cc:194] Registered new tserver with Master: 3fedff30870949fd9b5f03804fd5f5f6 (127.28.177.129:32897)
I20260808 02:17:28.225486 30200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:53859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:28.243894 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36365
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:28.330503 30484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:28.330705 30484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:28.330729 30484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:28.331823 30484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:28.331864 30484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:28.331892 30484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:28.333423 30484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:28.333637 30484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:28.333835 30484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:28.339314 30493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:28.339332 30489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:28.339305 30491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:28.339521 30484 server_base.cc:1034] running on GCE node
I20260808 02:17:28.339815 30484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:28.340031 30484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:28.341207 30484 hybrid_clock.cc:648] HybridClock initialized: now 1786155448341190 us; error 35 us; skew 500 ppm
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:28.343972 30484 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:28.344555 30484 webserver.cc:460] Webserver started at http://127.28.177.130:45387/ using document root <none> and password file <none>
I20260808 02:17:28.344828 30484 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:28.344959 30484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:28.345110 30484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:28.346155 30484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "0eac7574759a45d09bee16518d51372f"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "c985f5ea48180bc3e07270709048cafc"
server_key_iv: "82b7362d6263e66100d68bb1866f0725"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.346469 30484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "0eac7574759a45d09bee16518d51372f"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "c985f5ea48180bc3e07270709048cafc"
server_key_iv: "82b7362d6263e66100d68bb1866f0725"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.347822 30484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:28.348716 30504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.349050 30484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:28.349134 30484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "0eac7574759a45d09bee16518d51372f"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "c985f5ea48180bc3e07270709048cafc"
server_key_iv: "82b7362d6263e66100d68bb1866f0725"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.349201 30484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:28.363255 30484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:28.365764 30484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:28.365887 30484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:28.366088 30484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:28.366429 30484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:28.366467 30484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.366492 30484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:28.366539 30484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.372421 30484 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:39515
I20260808 02:17:28.372519 30663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:39515 every 8 connection(s)
I20260808 02:17:28.372891 30484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:28.380278 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 30484
I20260808 02:17:28.380386 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:28.380662 29382 external_mini_cluster.cc:1428] Setting key e3afdfc0623221e9ca585a5aba62e0d6
I20260808 02:17:28.383328 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.373707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:59939 (local address 127.28.177.190:36365)
0808 02:17:28.373796 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.373799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.373975 (+ 176us) server_negotiation.cc:407] Connection header received
0808 02:17:28.375165 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.375166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.375179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.375235 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.375898 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.376413 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.377536 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.377732 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.377792 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.380130 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:28.380132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:28.380133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:28.380143 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:28.381868 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.382283 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.382284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.382284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.382347 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.382602 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.382603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.382604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.382725 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:28.382800 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.383152 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.383236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:17:28.383697 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.373653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36365 (local address 127.28.177.130:59939)
0808 02:17:28.373843 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.373880 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.375051 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.375301 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.375304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.375571 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:28.375776 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.375780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.376543 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.376544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.377456 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.377459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.377846 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.377848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.377977 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.378504 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:28.378514 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:28.379993 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:28.381952 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:28.381959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:28.381962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:28.382195 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:28.382414 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:28.382415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:28.382416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:28.382523 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:28.382804 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:28.382811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:28.382951 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.383061 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.383243 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
I20260808 02:17:28.384483 30664 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36365
I20260808 02:17:28.384727 30664 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:28.384963 30664 heartbeater.cc:511] Master 127.28.177.190:36365 requested a full tablet report, sending...
I20260808 02:17:28.385404 30200 ts_manager.cc:194] Registered new tserver with Master: 0eac7574759a45d09bee16518d51372f (127.28.177.130:39515)
I20260808 02:17:28.385818 30200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:59939
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:28.412812 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36365
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:28.506618 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:28.506897 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:28.506947 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:28.508248 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:28.508317 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:28.508364 30671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:28.510058 30671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:28.510353 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:28.510612 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:28.516184 30691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:28.516199 30682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:28.516199 30683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:28.516229 30671 server_base.cc:1034] running on GCE node
I20260808 02:17:28.516695 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:28.517035 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:28.518204 30671 hybrid_clock.cc:648] HybridClock initialized: now 1786155448518180 us; error 38 us; skew 500 ppm
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:28.521116 30671 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:28.521600 30671 webserver.cc:460] Webserver started at http://127.28.177.131:42771/ using document root <none> and password file <none>
I20260808 02:17:28.521860 30671 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:28.521931 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:28.522092 30671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:28.523252 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "9bbb29c024b94325b3bba56dcb66621c"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "036e4abcac0174c90944d25d2be131e6"
server_key_iv: "99dce8ba24bff5cf43ae4d378b44cf08"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.523620 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "9bbb29c024b94325b3bba56dcb66621c"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "036e4abcac0174c90944d25d2be131e6"
server_key_iv: "99dce8ba24bff5cf43ae4d378b44cf08"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.525235 30671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:28.526089 30698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.526325 30671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:28.526438 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "9bbb29c024b94325b3bba56dcb66621c"
format_stamp: "Formatted at 2026-08-08 02:17:28 on dist-test-slave-3h5h"
server_key: "036e4abcac0174c90944d25d2be131e6"
server_key_iv: "99dce8ba24bff5cf43ae4d378b44cf08"
server_key_version: "encryptionkey@0"
I20260808 02:17:28.526513 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:28.561175 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:28.563407 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:28.563587 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:28.563845 30671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:28.564180 30671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:28.564237 30671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.564280 30671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:28.564320 30671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:28.570550 30671 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:41649
I20260808 02:17:28.570612 30865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:41649 every 8 connection(s)
I20260808 02:17:28.570930 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:28.579257 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 30671
I20260808 02:17:28.579402 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:28.579604 29382 external_mini_cluster.cc:1428] Setting key 29446096862b5ee3236ef87701cb1bcc
I20260808 02:17:28.582365 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.571813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:53329 (local address 127.28.177.190:36365)
0808 02:17:28.571924 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.571927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.572161 (+ 234us) server_negotiation.cc:407] Connection header received
0808 02:17:28.573337 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.573338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.573352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.573415 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.574181 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.574779 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.575993 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.576222 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.576325 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.578874 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:28.578875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:28.578876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:28.578884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:28.580786 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.581245 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.581246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.581246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.581304 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.581674 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.581675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.581675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.581791 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:28.581851 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.582220 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.582294 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:17:28.582453 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.571765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36365 (local address 127.28.177.131:53329)
0808 02:17:28.571966 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.572025 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.573212 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.573491 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.573496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.573851 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:28.574071 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.574076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.574899 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.574900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.575881 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.575885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.576339 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.576340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.576480 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.577165 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:28.577174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:28.578775 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:28.580893 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:28.580900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:28.580902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:28.581172 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:28.581469 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:28.581470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:28.581471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:28.581599 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:28.581875 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:28.581881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:28.582006 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.582111 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.582239 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260808 02:17:28.582990 30866 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36365
I20260808 02:17:28.583173 30866 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:28.583381 30866 heartbeater.cc:511] Master 127.28.177.190:36365 requested a full tablet report, sending...
I20260808 02:17:28.583858 30200 ts_manager.cc:194] Registered new tserver with Master: 9bbb29c024b94325b3bba56dcb66621c (127.28.177.131:41649)
I20260808 02:17:28.584396 30200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:53329
I20260808 02:17:28.591815 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:17:28.600538 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.594096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.28.177.190:36365)
0808 02:17:28.594185 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.594187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.594204 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:17:28.594269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.594269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.594280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.594332 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.594967 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.595468 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.596258 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.596449 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.596512 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.597880 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:28.597881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:28.597882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:28.597889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:28.599286 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.599611 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.599612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.599612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.599680 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.599897 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.599898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.599898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.600130 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:28.600218 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.600333 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.600402 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:17:28.603480 30200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:28.605415 30200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:28.612845 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.608497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:32897 (local address 127.0.0.1:34706)
0808 02:17:28.608824 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.608854 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.608909 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.609099 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.609103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.609106 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:28.609182 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.609185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.610724 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.610725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.611682 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.611685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.612536 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.612537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.612638 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.612653 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.612719 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.612761 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":88,"threads_started":1}
I20260808 02:17:28.612939 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.608655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.28.177.129:32897)
0808 02:17:28.608909 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.608914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.608925 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:28.608969 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.608971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.608994 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.609087 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:28.609277 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.610607 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.611749 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.612408 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.612523 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.612578 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.612767 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.612817 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260808 02:17:28.613279 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.608484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:39515 (local address 127.0.0.1:47304)
0808 02:17:28.608672 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.608719 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.608805 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.609310 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.609311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.609313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:28.609362 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.609364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.610443 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.610453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.611695 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.611698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.613024 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.613025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.613097 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.613107 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.613167 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.613206 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260808 02:17:28.613502 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.608708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:41649 (local address 127.0.0.1:40862)
0808 02:17:28.609191 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.609201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.609243 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.609549 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.609550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.609552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:28.609599 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.609600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.610903 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.610904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.611993 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.611995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.612924 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.612925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.613336 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.613341 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.613400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.613434 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":70,"threads_started":1}
I20260808 02:17:28.613505 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.608612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.28.177.130:39515)
0808 02:17:28.609034 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.609039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.609050 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:28.609095 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.609097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.609117 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.609209 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:28.609440 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.610335 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.611822 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.612413 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.612517 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.612579 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.613383 (+ 804us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.613429 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":99,"threads_started":1}
I20260808 02:17:28.613806 30377 tablet_service.cc:1467] Processing CreateTablet for tablet 224f9f0c57b94833b98e4c72773fc4cc (DEFAULT_TABLE table=test-table [id=ba6be0a6457340e7b9f2dd71d799f596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:28.614239 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f9f0c57b94833b98e4c72773fc4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:28.614181 30571 tablet_service.cc:1467] Processing CreateTablet for tablet 224f9f0c57b94833b98e4c72773fc4cc (DEFAULT_TABLE table=test-table [id=ba6be0a6457340e7b9f2dd71d799f596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:28.614512 30571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f9f0c57b94833b98e4c72773fc4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:28.617000 30888 tablet_bootstrap.cc:492] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: Bootstrap starting.
I20260808 02:17:28.617467 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.608926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.28.177.131:41649)
0808 02:17:28.609366 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.609371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.609380 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:28.609419 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.609421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.609436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.609669 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:28.609754 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.610794 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.612085 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.612786 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.612920 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.613134 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.617293 (+ 4159us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.617366 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":228,"threads_started":1}
I20260808 02:17:28.617715 30888 tablet_bootstrap.cc:654] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:28.618279 30888 log.cc:824] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:28.618245 30772 tablet_service.cc:1467] Processing CreateTablet for tablet 224f9f0c57b94833b98e4c72773fc4cc (DEFAULT_TABLE table=test-table [id=ba6be0a6457340e7b9f2dd71d799f596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:28.618634 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224f9f0c57b94833b98e4c72773fc4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:28.618662 30890 tablet_bootstrap.cc:492] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: Bootstrap starting.
I20260808 02:17:28.619150 30888 tablet_bootstrap.cc:492] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: No bootstrap required, opened a new log
I20260808 02:17:28.619223 30888 ts_tablet_manager.cc:1397] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:28.619344 30890 tablet_bootstrap.cc:654] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:28.619843 30890 log.cc:824] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:28.620699 30890 tablet_bootstrap.cc:492] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: No bootstrap required, opened a new log
I20260808 02:17:28.620779 30890 ts_tablet_manager.cc:1397] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:28.621569 30888 raft_consensus.cc:348] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.621771 30888 raft_consensus.cc:374] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:28.621814 30888 raft_consensus.cc:729] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eac7574759a45d09bee16518d51372f, State: Initialized, Role: FOLLOWER
I20260808 02:17:28.621961 30888 consensus_queue.cc:260] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.622402 30664 heartbeater.cc:503] Master 127.28.177.190:36365 was elected leader, sending a full tablet report...
I20260808 02:17:28.622596 30888 ts_tablet_manager.cc:1428] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:28.622824 30894 tablet_bootstrap.cc:492] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: Bootstrap starting.
I20260808 02:17:28.622921 30890 raft_consensus.cc:348] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.623116 30890 raft_consensus.cc:374] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:28.623170 30890 raft_consensus.cc:729] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fedff30870949fd9b5f03804fd5f5f6, State: Initialized, Role: FOLLOWER
I20260808 02:17:28.623296 30890 consensus_queue.cc:260] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
W20260808 02:17:28.623461 30667 tablet.cc:2369] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:28.623570 30890 ts_tablet_manager.cc:1428] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:28.623613 30894 tablet_bootstrap.cc:654] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:28.623761 30474 heartbeater.cc:503] Master 127.28.177.190:36365 was elected leader, sending a full tablet report...
I20260808 02:17:28.624202 30894 log.cc:824] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:28.625128 30894 tablet_bootstrap.cc:492] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: No bootstrap required, opened a new log
I20260808 02:17:28.625187 30894 ts_tablet_manager.cc:1397] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:28.627292 30894 raft_consensus.cc:348] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.627470 30894 raft_consensus.cc:374] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:28.627501 30894 raft_consensus.cc:729] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbb29c024b94325b3bba56dcb66621c, State: Initialized, Role: FOLLOWER
I20260808 02:17:28.627624 30894 consensus_queue.cc:260] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.627869 30894 ts_tablet_manager.cc:1428] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:28.627978 30866 heartbeater.cc:503] Master 127.28.177.190:36365 was elected leader, sending a full tablet report...
W20260808 02:17:28.711962 30475 tablet.cc:2369] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:28.821534 30867 tablet.cc:2369] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:28.925477 30899 raft_consensus.cc:482] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:28.925619 30899 raft_consensus.cc:504] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.926038 30899 leader_election.cc:288] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eac7574759a45d09bee16518d51372f (127.28.177.130:39515), 9bbb29c024b94325b3bba56dcb66621c (127.28.177.131:41649)
I20260808 02:17:28.929382 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.926367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:50153 (local address 127.28.177.130:39515)
0808 02:17:28.926442 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.926445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.926512 (+ 67us) server_negotiation.cc:407] Connection header received
0808 02:17:28.926597 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.926598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.926609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.926675 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:28.926906 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.927466 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.928604 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.929036 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.929094 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.929101 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.929261 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.929303 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:17:28.929394 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.926258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:39515 (local address 127.28.177.129:50153)
0808 02:17:28.926426 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.926441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.926533 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.926745 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.926747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.926751 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:28.926818 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.926821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.927565 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.927566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.928503 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.928506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.929120 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.929121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.929184 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.929189 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.929245 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.929277 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":25,"thread_start_us":86,"threads_started":1}
I20260808 02:17:28.929527 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.926580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:38127 (local address 127.28.177.131:41649)
0808 02:17:28.926625 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.926627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.926752 (+ 125us) server_negotiation.cc:407] Connection header received
0808 02:17:28.926779 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.926780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.926790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.926848 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:28.927132 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.927647 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.928660 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.929105 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.929179 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.929187 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.929408 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.929457 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:17:28.929596 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.926486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:41649 (local address 127.28.177.129:38127)
0808 02:17:28.926638 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:28.926648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:28.926699 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:28.926906 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:28.926907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:28.926910 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:28.927043 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.927046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.927739 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.927739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.928537 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:28.928540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.929189 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.929190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:28.929284 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.929288 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:28.929347 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:17:28.929387 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":14,"thread_start_us":75,"threads_started":1}
I20260808 02:17:28.929970 30601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f9f0c57b94833b98e4c72773fc4cc" candidate_uuid: "3fedff30870949fd9b5f03804fd5f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eac7574759a45d09bee16518d51372f" is_pre_election: true
I20260808 02:17:28.930197 30601 raft_consensus.cc:2454] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fedff30870949fd9b5f03804fd5f5f6 in term 0.
I20260808 02:17:28.930424 30316 leader_election.cc:302] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eac7574759a45d09bee16518d51372f, 3fedff30870949fd9b5f03804fd5f5f6; no voters:
I20260808 02:17:28.930590 30899 raft_consensus.cc:2781] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:28.930662 30899 raft_consensus.cc:482] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:28.930680 30899 raft_consensus.cc:3037] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:28.931450 30899 raft_consensus.cc:504] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.931602 30899 leader_election.cc:288] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [CANDIDATE]: Term 1 election: Requested vote from peers 0eac7574759a45d09bee16518d51372f (127.28.177.130:39515), 9bbb29c024b94325b3bba56dcb66621c (127.28.177.131:41649)
I20260808 02:17:28.931833 30601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f9f0c57b94833b98e4c72773fc4cc" candidate_uuid: "3fedff30870949fd9b5f03804fd5f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eac7574759a45d09bee16518d51372f"
I20260808 02:17:28.931838 30802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f9f0c57b94833b98e4c72773fc4cc" candidate_uuid: "3fedff30870949fd9b5f03804fd5f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbb29c024b94325b3bba56dcb66621c"
I20260808 02:17:28.931914 30601 raft_consensus.cc:3037] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:28.931977 30802 raft_consensus.cc:3037] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:28.929960 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224f9f0c57b94833b98e4c72773fc4cc" candidate_uuid: "3fedff30870949fd9b5f03804fd5f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbb29c024b94325b3bba56dcb66621c" is_pre_election: true
I20260808 02:17:28.932821 30802 raft_consensus.cc:2454] T 224f9f0c57b94833b98e4c72773fc4cc P 9bbb29c024b94325b3bba56dcb66621c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fedff30870949fd9b5f03804fd5f5f6 in term 1.
I20260808 02:17:28.933035 30601 raft_consensus.cc:2454] T 224f9f0c57b94833b98e4c72773fc4cc P 0eac7574759a45d09bee16518d51372f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fedff30870949fd9b5f03804fd5f5f6 in term 1.
I20260808 02:17:28.933089 30316 leader_election.cc:302] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fedff30870949fd9b5f03804fd5f5f6, 9bbb29c024b94325b3bba56dcb66621c; no voters:
I20260808 02:17:28.933235 30899 raft_consensus.cc:2781] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:28.933440 30899 raft_consensus.cc:686] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [term 1 LEADER]: Becoming Leader. State: Replica: 3fedff30870949fd9b5f03804fd5f5f6, State: Running, Role: LEADER
I20260808 02:17:28.933552 30899 consensus_queue.cc:237] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } }
I20260808 02:17:28.934513 30198 catalog_manager.cc:5495] T 224f9f0c57b94833b98e4c72773fc4cc P 3fedff30870949fd9b5f03804fd5f5f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fedff30870949fd9b5f03804fd5f5f6 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "3fedff30870949fd9b5f03804fd5f5f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac7574759a45d09bee16518d51372f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fedff30870949fd9b5f03804fd5f5f6" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 32897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bbb29c024b94325b3bba56dcb66621c" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41649 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:28.980049 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.971167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.28.177.129:32897)
0808 02:17:28.971299 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.971301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.971452 (+ 151us) server_negotiation.cc:407] Connection header received
0808 02:17:28.971536 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.971538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.971552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.971641 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.972441 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.973003 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.973805 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:28.974012 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:28.974107 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:28.976949 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:28.976974 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:28.976974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:28.976983 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:28.978646 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.979084 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.979089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.979089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.979163 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:28.979391 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:28.979393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:28.979393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:28.979579 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:28.979760 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:28.979831 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 02:17:28.979931 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:17:28 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:29.006875 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:28.998698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.28.177.129:32897)
0808 02:17:28.998769 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:28.998772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:28.998861 (+ 89us) server_negotiation.cc:407] Connection header received
0808 02:17:28.998912 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:28.998913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:28.998922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:28.998981 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:28.999612 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:29.000145 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:29.001095 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:29.001306 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:29.001372 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:29.003732 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:29.003734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:29.003734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:29.003742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:29.005378 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:29.005840 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:29.005842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:29.005842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:29.005915 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:29.006153 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:29.006154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:29.006155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:29.006276 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:29.006433 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:29.006658 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 02:17:29.006752 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 02:17:29 dist-test-slave-3h5h krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:17:29 dist-test-slave-3h5h krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:29.032471 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:29.025108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.28.177.129:32897)
0808 02:17:29.025195 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:29.025197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:29.025281 (+ 84us) server_negotiation.cc:407] Connection header received
0808 02:17:29.025347 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:29.025348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:29.025359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:29.025412 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:29.026030 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:29.026558 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:29.027302 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:29.027484 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:29.027538 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:29.030056 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:29.030057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:29.030058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:29.030066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:29.031425 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:29.031760 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:29.031760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:29.031761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:29.031816 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:29.031993 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:29.031994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:29.031994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:29.032093 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:29.032186 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:29.032286 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 02:17:29.032347 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 02:17:29.033244 30347 tablet_service.cc:2908] Rejecting scan request for tablet 224f9f0c57b94833b98e4c72773fc4cc: Uninitialized: clean time has not yet been initialized
I20260808 02:17:29.034281 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 30290
I20260808 02:17:29.046135 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 30484
I20260808 02:17:29.055147 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 30671
I20260808 02:17:29.067262 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 30164
2026-08-08T02:17:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2640 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:29 dist-test-slave-3h5h krb5kdc[30921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:29 dist-test-slave-3h5h krb5kdc[30921](info): set up 2 sockets
Aug 08 02:17:29 dist-test-slave-3h5h krb5kdc[30921](info): commencing operation
krb5kdc: starting...
W20260808 02:17:30.328401 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:30 dist-test-slave-3h5h krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:30.468385 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:34935
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:30.555272 30940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:30.555505 30940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:30.555528 30940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:30.556730 30940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:30.556773 30940 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:17:30.556787 30940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:30.556799 30940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:30.556881 30940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:30.556905 30940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:30.558447 30940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:34935
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:30.558663 30940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:30.558918 30940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:30.564419 30951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:30.564590 30940 server_base.cc:1034] running on GCE node
W20260808 02:17:30.564338 30945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:30.564342 30948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:30.565102 30940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:30.565454 30940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:30.566638 30940 hybrid_clock.cc:648] HybridClock initialized: now 1786155450566624 us; error 72 us; skew 500 ppm
Aug 08 02:17:30 dist-test-slave-3h5h krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:30.569578 30940 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:30.570115 30940 webserver.cc:460] Webserver started at http://127.28.177.190:35731/ using document root <none> and password file <none>
I20260808 02:17:30.570362 30940 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:30.570432 30940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:30.570623 30940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:30.571722 30940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "716952982294431bbccaef69c1824b49"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "fbbd0a772f25b19e4136f7fb131b0316"
server_key_iv: "d42f3a3c4e60e9cea6a337c78cd1303c"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.572103 30940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "716952982294431bbccaef69c1824b49"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "fbbd0a772f25b19e4136f7fb131b0316"
server_key_iv: "d42f3a3c4e60e9cea6a337c78cd1303c"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.573567 30940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:30.574453 30962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:30.574656 30940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:30.574769 30940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "716952982294431bbccaef69c1824b49"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "fbbd0a772f25b19e4136f7fb131b0316"
server_key_iv: "d42f3a3c4e60e9cea6a337c78cd1303c"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.574843 30940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:30.585268 30940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:30.587517 30940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:30.587738 30940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:30.593060 30940 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:34935
I20260808 02:17:30.593137 31051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:34935 every 8 connection(s)
I20260808 02:17:30.593495 30940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:30.594199 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 30940
I20260808 02:17:30.594306 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:30.594462 31054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:30.594547 29382 external_mini_cluster.cc:1428] Setting key d197205d050f9bb46b1cddd13931293c
I20260808 02:17:30.598253 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: Bootstrap starting.
I20260808 02:17:30.598961 31054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:30.599315 31054 log.cc:824] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:30.600033 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: No bootstrap required, opened a new log
Aug 08 02:17:30 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:30.602355 31054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716952982294431bbccaef69c1824b49" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34935 } }
I20260808 02:17:30.602552 31054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:30.602600 31054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716952982294431bbccaef69c1824b49, State: Initialized, Role: FOLLOWER
I20260808 02:17:30.602747 31054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716952982294431bbccaef69c1824b49" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34935 } }
I20260808 02:17:30.602837 31054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:30.602896 31054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:30.602980 31054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:30.603588 31054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716952982294431bbccaef69c1824b49" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34935 } }
I20260808 02:17:30.603718 31054 leader_election.cc:302] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 716952982294431bbccaef69c1824b49; no voters:
I20260808 02:17:30.603924 31054 leader_election.cc:288] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:30.603979 31064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:30.604135 31064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [term 1 LEADER]: Becoming Leader. State: Replica: 716952982294431bbccaef69c1824b49, State: Running, Role: LEADER
I20260808 02:17:30.604225 31054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:30.604272 31064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716952982294431bbccaef69c1824b49" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34935 } }
I20260808 02:17:30.604616 31068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "716952982294431bbccaef69c1824b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716952982294431bbccaef69c1824b49" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34935 } } }
I20260808 02:17:30.604782 31068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:30.604605 31064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 716952982294431bbccaef69c1824b49. Latest consensus state: current_term: 1 leader_uuid: "716952982294431bbccaef69c1824b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716952982294431bbccaef69c1824b49" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34935 } } }
I20260808 02:17:30.605131 31064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:30.605327 31074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:30.605851 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.595109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.28.177.190:34935)
0808 02:17:30.595285 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:30.595293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:30.595329 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:17:30.595717 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:30.595723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:30.595773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:30.595889 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:30.596724 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.597671 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.598743 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.599244 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.599329 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:30.601799 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:30.601804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:30.601805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:30.601818 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:30.603835 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:30.604309 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:30.604312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:30.604313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:30.604422 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:30.604672 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:30.604673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:30.604674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:30.605041 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:30.605184 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:30.605486 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:17:30.605653 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260808 02:17:30.605859 31074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:30.607486 31074 catalog_manager.cc:1347] Generated new cluster ID: 0d38545ce1354b05a791a205b4141560
I20260808 02:17:30.607563 31074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:30.627394 31074 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:30.628183 31074 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:30.639596 31074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: Generated new TSK 0
I20260808 02:17:30.639822 31074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:30.674165 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34935
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:30.784623 31091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:30.784911 31091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:30.784965 31091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:30.786198 31091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:30.786269 31091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:30.786304 31091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:17:30.786342 31091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:30.787866 31091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:34935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:30.788159 31091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:30.788444 31091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:30.793916 31098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:30.793906 31097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:30.793942 31101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:30.794389 31091 server_base.cc:1034] running on GCE node
I20260808 02:17:30.794598 31091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:30.794836 31091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:30.795986 31091 hybrid_clock.cc:648] HybridClock initialized: now 1786155450796022 us; error 81 us; skew 500 ppm
Aug 08 02:17:30 dist-test-slave-3h5h krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:30.798918 31091 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:30.799506 31091 webserver.cc:460] Webserver started at http://127.28.177.129:39273/ using document root <none> and password file <none>
I20260808 02:17:30.799747 31091 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:30.799817 31091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:30.799975 31091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:30.801213 31091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "5bbf68dd26db4b209f8aaedcf72f2675"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "1f9a797eb46f3ce38f82f0c676043a0c"
server_key_iv: "008299e8b92ba85c73eb0af9e68f00bf"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.801748 31091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "5bbf68dd26db4b209f8aaedcf72f2675"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "1f9a797eb46f3ce38f82f0c676043a0c"
server_key_iv: "008299e8b92ba85c73eb0af9e68f00bf"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.803229 31091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:30.804209 31109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:30.804410 31091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:30.804522 31091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "5bbf68dd26db4b209f8aaedcf72f2675"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "1f9a797eb46f3ce38f82f0c676043a0c"
server_key_iv: "008299e8b92ba85c73eb0af9e68f00bf"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.804620 31091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:30.820448 31091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:30.823137 31091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:30.823331 31091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:30.823983 31091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:30.824050 31091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:30.824101 31091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:30.824131 31091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:17:30 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:30.831898 31091 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:34281
I20260808 02:17:30.831998 31277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:34281 every 8 connection(s)
I20260808 02:17:30.832317 31091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:30.834777 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.824762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:39559 (local address 127.28.177.190:34935)
0808 02:17:30.824867 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:30.824869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:30.825108 (+ 239us) server_negotiation.cc:407] Connection header received
0808 02:17:30.825659 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:30.825660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:30.825675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:30.825724 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:30.826485 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.827030 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.828164 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.828400 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.828524 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:30.831703 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:30.831705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:30.831705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:30.831713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:30.833413 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:30.833853 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:30.833854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:30.833854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:30.833914 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:30.834200 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:30.834201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:30.834201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:30.834320 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:30.834377 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:30.834634 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 02:17:30.834711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:17:30.834934 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.824715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.28.177.129:39559)
0808 02:17:30.824935 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:30.824975 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:30.825537 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:30.825812 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:30.825816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:30.826143 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:30.826348 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:30.826353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.827141 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.827142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:30.828066 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:30.828069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.828536 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.828537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:30.828692 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:30.829402 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:30.829412 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:30.831587 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:30.833529 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:30.833539 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:30.833541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:30.833778 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:30.834001 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:30.834002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:30.834003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:30.834121 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:30.834384 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:30.834390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:30.834515 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:17:30.834603 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:17:30.834731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":124,"threads_started":1}
I20260808 02:17:30.835485 31280 heartbeater.cc:347] Connected to a master server at 127.28.177.190:34935
I20260808 02:17:30.835637 31280 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:30.835837 31280 heartbeater.cc:511] Master 127.28.177.190:34935 requested a full tablet report, sending...
I20260808 02:17:30.836427 30984 ts_manager.cc:194] Registered new tserver with Master: 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281)
I20260808 02:17:30.837203 30984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:39559
I20260808 02:17:30.841202 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 31091
I20260808 02:17:30.841288 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:30.841493 29382 external_mini_cluster.cc:1428] Setting key 35b053549e4516c9a5a8daec5c2e1026
I20260808 02:17:30.842633 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.836753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:56901 (local address 127.28.177.190:34935)
0808 02:17:30.836810 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:30.836811 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:30.836967 (+ 156us) server_negotiation.cc:407] Connection header received
0808 02:17:30.837051 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:30.837052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:30.837062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:30.837112 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:30.837587 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.838047 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.838764 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.838946 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.839021 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:30.839946 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:30.839948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:30.839948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:30.839955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:30.841385 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:30.841815 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:30.841817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:30.841817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:30.841882 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:30.842093 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:30.842094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:30.842095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:30.842206 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:30.842282 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:30.842461 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:17:30.842540 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:17:30.842640 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.836690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.28.177.129:56901)
0808 02:17:30.836889 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:30.836899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:30.836981 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:30.837114 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:30.837115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:30.837332 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:30.837401 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:30.837403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.838160 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.838161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:30.838679 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:30.838680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:30.839037 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:30.839038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:30.839098 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:30.839534 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:30.839541 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:30.839850 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:30.841448 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:30.841449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:30.841449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:30.841703 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:30.841953 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:30.841954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:30.841954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:30.842020 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:30.842286 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:30.842287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:30.842336 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:17:30.842419 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:17:30.842542 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:30.873135 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34935
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:30.959177 31296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:30.959434 31296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:30.959486 31296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:30.960708 31296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:30.960779 31296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:30.960815 31296 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:17:30.960877 31296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:30.962376 31296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:34935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:30.962669 31296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:30.962909 31296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:30.968530 31301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:30.968544 31302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:30.968530 31309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:30.968544 31296 server_base.cc:1034] running on GCE node
I20260808 02:17:30.969172 31296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:30.969481 31296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:30.970655 31296 hybrid_clock.cc:648] HybridClock initialized: now 1786155450970652 us; error 58 us; skew 500 ppm
Aug 08 02:17:30 dist-test-slave-3h5h krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:30.973318 31296 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:30.973807 31296 webserver.cc:460] Webserver started at http://127.28.177.130:39723/ using document root <none> and password file <none>
I20260808 02:17:30.974081 31296 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:30.974130 31296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:30.974289 31296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:30.975389 31296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "13632200ba98416f81684518fe763abd"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "51dcdeef6cd0b1a53d71b686e71cb3fc"
server_key_iv: "66c8b01db18c1105e08457a1f18192cb"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.975775 31296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "13632200ba98416f81684518fe763abd"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "51dcdeef6cd0b1a53d71b686e71cb3fc"
server_key_iv: "66c8b01db18c1105e08457a1f18192cb"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.977262 31296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:30.978124 31320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:30.978322 31296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:30.978417 31296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "13632200ba98416f81684518fe763abd"
format_stamp: "Formatted at 2026-08-08 02:17:30 on dist-test-slave-3h5h"
server_key: "51dcdeef6cd0b1a53d71b686e71cb3fc"
server_key_iv: "66c8b01db18c1105e08457a1f18192cb"
server_key_version: "encryptionkey@0"
I20260808 02:17:30.978492 31296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:30.995226 31296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:30.997454 31296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:30.997568 31296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:30.998159 31296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:30.998211 31296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:30.998246 31296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:30.998265 31296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:31.006137 31296 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:33235
I20260808 02:17:31.006237 31477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:33235 every 8 connection(s)
I20260808 02:17:31.006641 31296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:17:31.009527 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 31296
I20260808 02:17:31.009634 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:31.009809 29382 external_mini_cluster.cc:1428] Setting key 7bf6f4c546fa9b8f175b9caccd3699d6
I20260808 02:17:31.009816 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.998642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:48303 (local address 127.28.177.190:34935)
0808 02:17:30.998720 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:30.998723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:30.998995 (+ 272us) server_negotiation.cc:407] Connection header received
0808 02:17:30.999763 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:30.999766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:30.999782 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:30.999867 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.000816 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.001423 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.003175 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.003423 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.003524 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.006554 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.006555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.006556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.006563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.008274 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.008723 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.008724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.008725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.008789 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.009130 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.009131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.009132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.009265 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.009342 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.009634 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.009723 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:31.010011 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:30.998603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.28.177.130:48303)
0808 02:17:30.998832 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:30.998891 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:30.999638 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:30.999955 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:30.999961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.000364 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:31.000690 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.000698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.001521 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.001523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.003065 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.003069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.003543 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.003545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.003722 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.004346 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:31.004369 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:31.006458 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:31.008370 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.008377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.008380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.008615 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.008905 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.008905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.008906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.009049 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.009343 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:31.009351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:31.009498 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.009610 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.009777 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260808 02:17:31.010629 31478 heartbeater.cc:347] Connected to a master server at 127.28.177.190:34935
I20260808 02:17:31.010720 31478 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:31.010957 31478 heartbeater.cc:511] Master 127.28.177.190:34935 requested a full tablet report, sending...
I20260808 02:17:31.011376 30983 ts_manager.cc:194] Registered new tserver with Master: 13632200ba98416f81684518fe763abd (127.28.177.130:33235)
I20260808 02:17:31.011914 30983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:48303
I20260808 02:17:31.018452 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.011720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:47183 (local address 127.28.177.190:34935)
0808 02:17:31.011763 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.011765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.011931 (+ 166us) server_negotiation.cc:407] Connection header received
0808 02:17:31.012029 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.012030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.012042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.012073 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.012550 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.013161 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.013959 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.014153 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.014213 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.015326 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.015327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.015328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.015334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.017035 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.017526 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.017528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.017528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.017600 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.017838 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.017839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.017839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.018007 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.018074 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.018281 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.018355 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:17:31.018496 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.011663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.28.177.130:47183)
0808 02:17:31.011862 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.011873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.011957 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.012134 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.012135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.012352 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:31.012421 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.012423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.013259 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.013260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.013871 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.013873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.014236 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.014237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.014322 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.014756 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:31.014762 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:31.015218 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:31.017128 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.017129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.017130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.017386 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.017688 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.017689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.017689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.017763 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.018087 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:31.018089 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:31.018150 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.018239 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.018355 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":148,"threads_started":1}
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:31.040498 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34935
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:31.127964 31487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:31.128224 31487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:31.128265 31487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:31.129640 31487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:31.129716 31487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:31.129751 31487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:17:31.129791 31487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:31.131317 31487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:34935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:31.131606 31487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:31.131847 31487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:31.137329 31494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:31.137359 31498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:31.137419 31495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:31.137677 31487 server_base.cc:1034] running on GCE node
I20260808 02:17:31.137929 31487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:31.138206 31487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:31.139397 31487 hybrid_clock.cc:648] HybridClock initialized: now 1786155451139374 us; error 35 us; skew 500 ppm
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155451, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:31.142532 31487 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:31.143043 31487 webserver.cc:460] Webserver started at http://127.28.177.131:42863/ using document root <none> and password file <none>
I20260808 02:17:31.143342 31487 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:31.143420 31487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:31.143589 31487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:31.144779 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "34396a22f4664b5d91623197749a36ac"
format_stamp: "Formatted at 2026-08-08 02:17:31 on dist-test-slave-3h5h"
server_key: "b03d85697625584b439151ac19f9496e"
server_key_iv: "bbc807089591ef9520f5089491550745"
server_key_version: "encryptionkey@0"
I20260808 02:17:31.145254 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "34396a22f4664b5d91623197749a36ac"
format_stamp: "Formatted at 2026-08-08 02:17:31 on dist-test-slave-3h5h"
server_key: "b03d85697625584b439151ac19f9496e"
server_key_iv: "bbc807089591ef9520f5089491550745"
server_key_version: "encryptionkey@0"
I20260808 02:17:31.146826 31487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:31.147804 31505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:31.148084 31487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:31.148221 31487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "34396a22f4664b5d91623197749a36ac"
format_stamp: "Formatted at 2026-08-08 02:17:31 on dist-test-slave-3h5h"
server_key: "b03d85697625584b439151ac19f9496e"
server_key_iv: "bbc807089591ef9520f5089491550745"
server_key_version: "encryptionkey@0"
I20260808 02:17:31.148305 31487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:31.170022 31487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:31.172183 31487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:31.172379 31487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:31.173166 31487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:31.173233 31487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:31.173285 31487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:31.173314 31487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155451, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:31.181785 31487 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:44575
I20260808 02:17:31.181936 31668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:44575 every 8 connection(s)
I20260808 02:17:31.182279 31487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:17:31.184243 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.173524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:40157 (local address 127.28.177.190:34935)
0808 02:17:31.173603 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.173605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.173910 (+ 305us) server_negotiation.cc:407] Connection header received
0808 02:17:31.174503 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.174505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.174522 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.174619 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.175411 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.176362 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.177992 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.178375 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.178465 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.180855 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.180857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.180858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.180866 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.182799 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.183351 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.183352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.183352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.183425 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.183631 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.183632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.183632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.183749 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.183810 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.184095 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.184176 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:17:31.184355 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.173497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.28.177.131:40157)
0808 02:17:31.173758 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.173802 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.174372 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.174672 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.174675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.174980 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:31.175298 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.175305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.176469 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.176470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.177892 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.177896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.178482 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.178483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.178637 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.179259 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:31.179269 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:31.180728 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:31.182922 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.182929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.182931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.183272 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.183484 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.183484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.183485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.183575 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.183801 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:31.183803 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:31.183929 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.184017 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.184142 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":139,"threads_started":1}
I20260808 02:17:31.184901 31669 heartbeater.cc:347] Connected to a master server at 127.28.177.190:34935
I20260808 02:17:31.185025 31669 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:31.185211 31669 heartbeater.cc:511] Master 127.28.177.190:34935 requested a full tablet report, sending...
I20260808 02:17:31.185670 30984 ts_manager.cc:194] Registered new tserver with Master: 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575)
I20260808 02:17:31.186169 30984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:40157
I20260808 02:17:31.187112 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 31487
I20260808 02:17:31.187215 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:31.187383 29382 external_mini_cluster.cc:1428] Setting key 9a17af435c0f726169bb7b8633d36344
I20260808 02:17:31.188280 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:17:31.193017 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.186060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:54143 (local address 127.28.177.190:34935)
0808 02:17:31.186099 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.186101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.186166 (+ 65us) server_negotiation.cc:407] Connection header received
0808 02:17:31.186291 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.186292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.186302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.186359 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.186775 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.187372 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.188450 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.188639 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.188697 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.189936 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.189938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.189938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.189944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.191457 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.191925 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.191926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.191927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.192083 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.192366 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.192367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.192368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.192506 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.192583 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.192767 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.192914 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:17:31.193046 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.185957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.28.177.131:54143)
0808 02:17:31.186104 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.186116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.186180 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.186406 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.186407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.186627 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:31.186692 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.186693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.187503 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.187504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.188333 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.188336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.188722 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.188724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.188817 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.189396 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:31.189403 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:31.189829 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:31.191546 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.191547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.191548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.191811 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.192172 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.192173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.192174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.192250 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.192592 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:31.192594 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:31.192650 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.192727 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.192881 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:31.214404 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.206893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.28.177.190:34935)
0808 02:17:31.206969 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.206971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.206982 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:31.207058 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.207059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.207075 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.207170 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.207781 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.208295 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.209113 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.209309 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.209387 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.211780 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.211782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.211782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.211790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.213238 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.213608 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.213609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.213609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.213661 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.213915 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.213916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.213916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.214045 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.214142 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.214268 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.214333 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:17:31.216271 30984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:31.218150 30984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:31.225173 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.220697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:34281 (local address 127.0.0.1:41526)
0808 02:17:31.220997 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.221032 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.221089 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.221595 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.221596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.221598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.221658 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.221661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.222590 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.222592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.223810 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.223812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.224825 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.224826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.224942 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.224959 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.225035 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.225090 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":67,"threads_started":1}
I20260808 02:17:31.225266 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.221025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.28.177.129:34281)
0808 02:17:31.221220 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.221225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.221237 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:17:31.221279 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.221281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.221298 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.221744 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.221807 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.222496 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.223890 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.224723 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.224817 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.224911 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.225078 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.225126 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260808 02:17:31.225499 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.220696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44575 (local address 127.0.0.1:42430)
0808 02:17:31.220813 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.220876 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.220970 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.221498 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.221502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.221504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.221609 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.221612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.222994 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.222995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.224029 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.224031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.225224 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.225226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.225306 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.225310 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.225351 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.225408 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":98,"threads_started":1}
I20260808 02:17:31.226001 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.221110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.28.177.131:44575)
0808 02:17:31.221301 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.221305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.221317 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:17:31.221357 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.221360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.221375 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.221700 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.221770 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.222897 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.224118 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.225107 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.225587 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.225670 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.225703 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.225748 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":124,"threads_started":1}
I20260808 02:17:31.226080 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.220697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:33235 (local address 127.0.0.1:49934)
0808 02:17:31.221334 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.221411 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.221459 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.221709 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.221709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.221711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.221750 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.221751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.224111 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.224112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.224798 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.224800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.225792 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.225793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.225861 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.225866 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.225944 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.225987 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260808 02:17:31.226102 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.221284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.28.177.130:33235)
0808 02:17:31.221487 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.221491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.221501 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:31.221537 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.221539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.221553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.221644 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.221832 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.222905 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.224917 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.225717 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.225794 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.225847 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.225958 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.225998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260808 02:17:31.226392 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 9eacdf6082c943408039b557ca2ecb32 (DEFAULT_TABLE table=test-table [id=f48158e898ee42bca0b938b6fbb3a4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:31.226832 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eacdf6082c943408039b557ca2ecb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.226754 31579 tablet_service.cc:1467] Processing CreateTablet for tablet 9eacdf6082c943408039b557ca2ecb32 (DEFAULT_TABLE table=test-table [id=f48158e898ee42bca0b938b6fbb3a4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:31.227015 31389 tablet_service.cc:1467] Processing CreateTablet for tablet 9eacdf6082c943408039b557ca2ecb32 (DEFAULT_TABLE table=test-table [id=f48158e898ee42bca0b938b6fbb3a4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:31.227156 31579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eacdf6082c943408039b557ca2ecb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.227371 31389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eacdf6082c943408039b557ca2ecb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.229950 31701 tablet_bootstrap.cc:492] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Bootstrap starting.
I20260808 02:17:31.230700 31701 tablet_bootstrap.cc:654] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.230927 31702 tablet_bootstrap.cc:492] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Bootstrap starting.
I20260808 02:17:31.231137 31701 log.cc:824] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:31.231874 31702 tablet_bootstrap.cc:654] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.231889 31704 tablet_bootstrap.cc:492] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Bootstrap starting.
I20260808 02:17:31.232035 31701 tablet_bootstrap.cc:492] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: No bootstrap required, opened a new log
I20260808 02:17:31.232131 31701 ts_tablet_manager.cc:1397] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:17:31.232288 31702 log.cc:824] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:31.232700 31704 tablet_bootstrap.cc:654] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.233107 31704 log.cc:824] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:31.233129 31702 tablet_bootstrap.cc:492] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: No bootstrap required, opened a new log
I20260808 02:17:31.233212 31702 ts_tablet_manager.cc:1397] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:31.234282 31701 raft_consensus.cc:348] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.234465 31701 raft_consensus.cc:374] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.234503 31701 raft_consensus.cc:729] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbf68dd26db4b209f8aaedcf72f2675, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.234644 31701 consensus_queue.cc:260] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.234920 31701 ts_tablet_manager.cc:1428] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:31.234970 31280 heartbeater.cc:503] Master 127.28.177.190:34935 was elected leader, sending a full tablet report...
I20260808 02:17:31.235522 31702 raft_consensus.cc:348] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.235680 31704 tablet_bootstrap.cc:492] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: No bootstrap required, opened a new log
I20260808 02:17:31.235682 31702 raft_consensus.cc:374] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.235725 31702 raft_consensus.cc:729] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34396a22f4664b5d91623197749a36ac, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.235786 31704 ts_tablet_manager.cc:1397] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 02:17:31.235913 31702 consensus_queue.cc:260] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.236217 31669 heartbeater.cc:503] Master 127.28.177.190:34935 was elected leader, sending a full tablet report...
I20260808 02:17:31.236199 31702 ts_tablet_manager.cc:1428] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:17:31.237753 31704 raft_consensus.cc:348] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.237875 31704 raft_consensus.cc:374] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.237896 31704 raft_consensus.cc:729] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13632200ba98416f81684518fe763abd, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.238027 31704 consensus_queue.cc:260] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.238305 31704 ts_tablet_manager.cc:1428] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:31.238376 31478 heartbeater.cc:503] Master 127.28.177.190:34935 was elected leader, sending a full tablet report...
I20260808 02:17:31.239192 31710 raft_consensus.cc:482] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:31.239256 31710 raft_consensus.cc:504] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.239565 31710 leader_election.cc:288] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13632200ba98416f81684518fe763abd (127.28.177.130:33235), 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281)
I20260808 02:17:31.242596 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.239933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:55357 (local address 127.28.177.130:33235)
0808 02:17:31.239970 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.239971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.239978 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:17:31.240014 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.240015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.240054 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.240125 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.240283 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.240753 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.241754 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.242219 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.242304 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.242314 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.242489 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.242523 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:17:31.242612 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.239737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:33235 (local address 127.28.177.131:55357)
0808 02:17:31.239791 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.239804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.239852 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.240143 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.240144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.240146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.240208 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.240210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.240863 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.240864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.241662 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.241664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.242300 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.242301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.242400 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.242403 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.242480 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.242521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 02:17:31.242647 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.239869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:40089 (local address 127.28.177.129:34281)
0808 02:17:31.239914 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.239916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.239990 (+ 74us) server_negotiation.cc:407] Connection header received
0808 02:17:31.240053 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.240054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.240067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.240126 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.240307 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.240780 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.241754 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.242257 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.242355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.242363 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.242535 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.242576 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:17:31.242697 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.239746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:34281 (local address 127.28.177.131:40089)
0808 02:17:31.239908 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.239918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.240002 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.240161 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.240162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.240164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.240239 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.240240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.240889 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.240890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.241661 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.241663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.242357 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.242358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.242447 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.242450 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.242526 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.242566 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":33,"thread_start_us":81,"threads_started":1}
I20260808 02:17:31.242954 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eacdf6082c943408039b557ca2ecb32" candidate_uuid: "34396a22f4664b5d91623197749a36ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13632200ba98416f81684518fe763abd" is_pre_election: true
I20260808 02:17:31.243095 31418 raft_consensus.cc:2454] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34396a22f4664b5d91623197749a36ac in term 0.
I20260808 02:17:31.243193 31212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eacdf6082c943408039b557ca2ecb32" candidate_uuid: "34396a22f4664b5d91623197749a36ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" is_pre_election: true
I20260808 02:17:31.243319 31507 leader_election.cc:302] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13632200ba98416f81684518fe763abd, 34396a22f4664b5d91623197749a36ac; no voters:
I20260808 02:17:31.243398 31212 raft_consensus.cc:2454] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34396a22f4664b5d91623197749a36ac in term 0.
I20260808 02:17:31.243464 31710 raft_consensus.cc:2781] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:31.243515 31710 raft_consensus.cc:482] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:31.243546 31710 raft_consensus.cc:3037] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.244261 31710 raft_consensus.cc:504] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.244366 31710 leader_election.cc:288] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [CANDIDATE]: Term 1 election: Requested vote from peers 13632200ba98416f81684518fe763abd (127.28.177.130:33235), 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281)
I20260808 02:17:31.244551 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eacdf6082c943408039b557ca2ecb32" candidate_uuid: "34396a22f4664b5d91623197749a36ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13632200ba98416f81684518fe763abd"
I20260808 02:17:31.244556 31212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eacdf6082c943408039b557ca2ecb32" candidate_uuid: "34396a22f4664b5d91623197749a36ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbf68dd26db4b209f8aaedcf72f2675"
I20260808 02:17:31.244620 31418 raft_consensus.cc:3037] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.244653 31212 raft_consensus.cc:3037] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.245429 31418 raft_consensus.cc:2454] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34396a22f4664b5d91623197749a36ac in term 1.
I20260808 02:17:31.245515 31212 raft_consensus.cc:2454] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34396a22f4664b5d91623197749a36ac in term 1.
I20260808 02:17:31.245626 31507 leader_election.cc:302] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13632200ba98416f81684518fe763abd, 34396a22f4664b5d91623197749a36ac; no voters:
I20260808 02:17:31.245743 31710 raft_consensus.cc:2781] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:31.245927 31710 raft_consensus.cc:686] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 1 LEADER]: Becoming Leader. State: Replica: 34396a22f4664b5d91623197749a36ac, State: Running, Role: LEADER
I20260808 02:17:31.246029 31710 consensus_queue.cc:237] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } }
I20260808 02:17:31.246847 30984 catalog_manager.cc:5495] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 34396a22f4664b5d91623197749a36ac (127.28.177.131). New cstate: current_term: 1 leader_uuid: "34396a22f4664b5d91623197749a36ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } health_report { overall_health: UNKNOWN } } }
W20260808 02:17:31.257378 31479 tablet.cc:2369] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:31.260092 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.257119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.28.177.131:44575)
0808 02:17:31.257183 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.257190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.257198 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:31.257230 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.257231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.257267 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.257349 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:31.257493 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.258043 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.258887 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.259281 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.259360 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.259563 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.259660 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.259725 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.259780 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.259801 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:17:31.262162 31212 raft_consensus.cc:1261] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Refusing update from remote peer 34396a22f4664b5d91623197749a36ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:31.262162 31418 raft_consensus.cc:1261] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Refusing update from remote peer 34396a22f4664b5d91623197749a36ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:31.262609 31710 consensus_queue.cc:1035] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:31.262766 31715 consensus_queue.cc:1035] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:31.269846 30983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:31.269997 30983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:31.270874 30983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: Sending DeleteTablet for 3 replicas of tablet 9eacdf6082c943408039b557ca2ecb32
I20260808 02:17:31.271303 31389 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eacdf6082c943408039b557ca2ecb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:49934
I20260808 02:17:31.271323 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eacdf6082c943408039b557ca2ecb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:41526
I20260808 02:17:31.271588 31732 tablet_replica.cc:330] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: stopping tablet replica
I20260808 02:17:31.271703 31732 raft_consensus.cc:2229] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:31.271699 31579 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eacdf6082c943408039b557ca2ecb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42430
I20260808 02:17:31.271931 31733 tablet_replica.cc:330] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: stopping tablet replica
I20260808 02:17:31.271909 30983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:17:31.272066 31733 raft_consensus.cc:2229] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 1 LEADER]: Raft consensus shutting down.
W20260808 02:17:31.272089 30983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:31.272104 31732 raft_consensus.cc:2258] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:31.272171 31733 raft_consensus.cc:2258] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:31.272776 31733 ts_tablet_manager.cc:1905] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:31.272666 31732 ts_tablet_manager.cc:1905] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:31.273588 31389 tablet_service.cc:1467] Processing CreateTablet for tablet 78edf709ed8f4411a6559f5451295eb9 (DEFAULT_TABLE table=test-table [id=16647ce32a9945c485a0f08f02d7c160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:31.273715 31389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edf709ed8f4411a6559f5451295eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.274883 31704 tablet_bootstrap.cc:492] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Bootstrap starting.
I20260808 02:17:31.275007 31732 ts_tablet_manager.cc:1918] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:31.275076 31732 log.cc:1196] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/9eacdf6082c943408039b557ca2ecb32
I20260808 02:17:31.275311 31704 tablet_bootstrap.cc:654] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.275380 31732 ts_tablet_manager.cc:1939] T 9eacdf6082c943408039b557ca2ecb32 P 13632200ba98416f81684518fe763abd: Deleting consensus metadata
I20260808 02:17:31.275553 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 78edf709ed8f4411a6559f5451295eb9 (DEFAULT_TABLE table=test-table [id=16647ce32a9945c485a0f08f02d7c160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:31.275664 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edf709ed8f4411a6559f5451295eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.275321 31579 tablet_service.cc:1467] Processing CreateTablet for tablet 78edf709ed8f4411a6559f5451295eb9 (DEFAULT_TABLE table=test-table [id=16647ce32a9945c485a0f08f02d7c160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:31.275808 30964 catalog_manager.cc:4841] TS 13632200ba98416f81684518fe763abd (127.28.177.130:33235): tablet 9eacdf6082c943408039b557ca2ecb32 (table test-table [id=f48158e898ee42bca0b938b6fbb3a4c3]) successfully deleted
I20260808 02:17:31.275913 31579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edf709ed8f4411a6559f5451295eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.276013 31731 tablet_replica.cc:330] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: stopping tablet replica
I20260808 02:17:31.276116 31731 raft_consensus.cc:2229] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:31.276216 31731 raft_consensus.cc:2258] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:31.276769 31731 ts_tablet_manager.cc:1905] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:31.277359 31702 tablet_bootstrap.cc:492] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Bootstrap starting.
I20260808 02:17:31.274829 31733 ts_tablet_manager.cc:1918] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:31.277441 31701 tablet_bootstrap.cc:492] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Bootstrap starting.
I20260808 02:17:31.277459 31733 log.cc:1196] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/9eacdf6082c943408039b557ca2ecb32
I20260808 02:17:31.277765 31733 ts_tablet_manager.cc:1939] T 9eacdf6082c943408039b557ca2ecb32 P 34396a22f4664b5d91623197749a36ac: Deleting consensus metadata
I20260808 02:17:31.277848 31702 tablet_bootstrap.cc:654] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.277878 31701 tablet_bootstrap.cc:654] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.278302 30963 catalog_manager.cc:4841] TS 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575): tablet 9eacdf6082c943408039b557ca2ecb32 (table test-table [id=f48158e898ee42bca0b938b6fbb3a4c3]) successfully deleted
I20260808 02:17:31.278674 31701 tablet_bootstrap.cc:492] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: No bootstrap required, opened a new log
I20260808 02:17:31.278735 31701 ts_tablet_manager.cc:1397] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:31.278887 31701 raft_consensus.cc:348] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.278966 31701 raft_consensus.cc:374] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.278988 31701 raft_consensus.cc:729] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbf68dd26db4b209f8aaedcf72f2675, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.279043 31701 consensus_queue.cc:260] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.279145 31701 ts_tablet_manager.cc:1428] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:31.279420 31731 ts_tablet_manager.cc:1918] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:31.279498 31702 tablet_bootstrap.cc:492] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: No bootstrap required, opened a new log
I20260808 02:17:31.279565 31702 ts_tablet_manager.cc:1397] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:17:31.279543 31731 log.cc:1196] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/9eacdf6082c943408039b557ca2ecb32
I20260808 02:17:31.279723 31702 raft_consensus.cc:348] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.279796 31702 raft_consensus.cc:374] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.279819 31702 raft_consensus.cc:729] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34396a22f4664b5d91623197749a36ac, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.279877 31702 consensus_queue.cc:260] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.279969 31702 ts_tablet_manager.cc:1428] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:31.280175 31731 ts_tablet_manager.cc:1939] T 9eacdf6082c943408039b557ca2ecb32 P 5bbf68dd26db4b209f8aaedcf72f2675: Deleting consensus metadata
I20260808 02:17:31.280673 30965 catalog_manager.cc:4841] TS 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281): tablet 9eacdf6082c943408039b557ca2ecb32 (table test-table [id=f48158e898ee42bca0b938b6fbb3a4c3]) successfully deleted
I20260808 02:17:31.281234 31704 tablet_bootstrap.cc:492] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: No bootstrap required, opened a new log
I20260808 02:17:31.281293 31704 ts_tablet_manager.cc:1397] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260808 02:17:31.281495 31704 raft_consensus.cc:348] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.281594 31704 raft_consensus.cc:374] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.281618 31704 raft_consensus.cc:729] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13632200ba98416f81684518fe763abd, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.281668 31704 consensus_queue.cc:260] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.281756 31704 ts_tablet_manager.cc:1428] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260808 02:17:31.333133 31281 tablet.cc:2369] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:31.432942 31670 tablet.cc:2369] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:31.505470 31711 raft_consensus.cc:482] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:31.505599 31711 raft_consensus.cc:504] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.506278 31711 leader_election.cc:288] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281), 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575)
I20260808 02:17:31.510210 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.506687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:33061 (local address 127.28.177.129:34281)
0808 02:17:31.506763 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.506765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.506776 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:31.506808 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.506809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.506837 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.506892 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.507185 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.507768 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.509269 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.509817 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.509908 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.509918 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.510090 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.510134 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:17:31.510210 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.506547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:45681 (local address 127.28.177.131:44575)
0808 02:17:31.506609 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.506612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.506842 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:17:31.506889 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.506890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.506905 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.506978 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.507255 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.507953 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.509268 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.509847 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.509913 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.509918 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.510099 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.510135 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:17:31.510258 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.506456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:34281 (local address 127.28.177.130:33061)
0808 02:17:31.506526 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.506539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.506592 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.507031 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.507032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.507034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.507103 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.507105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.507876 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.507877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.509176 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.509179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.509923 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.509924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.510005 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.510009 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.510070 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.510143 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":6}
I20260808 02:17:31.510272 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.506456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44575 (local address 127.28.177.130:45681)
0808 02:17:31.506671 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.506681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.506806 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.506991 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.506992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.506996 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.507106 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.507108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.508027 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.508028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.509180 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.509182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.509934 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.509935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.510007 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.510010 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.510072 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.510143 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260808 02:17:31.510664 31212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edf709ed8f4411a6559f5451295eb9" candidate_uuid: "13632200ba98416f81684518fe763abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" is_pre_election: true
I20260808 02:17:31.510800 31212 raft_consensus.cc:2454] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13632200ba98416f81684518fe763abd in term 0.
I20260808 02:17:31.510787 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edf709ed8f4411a6559f5451295eb9" candidate_uuid: "13632200ba98416f81684518fe763abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34396a22f4664b5d91623197749a36ac" is_pre_election: true
I20260808 02:17:31.510941 31604 raft_consensus.cc:2454] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13632200ba98416f81684518fe763abd in term 0.
I20260808 02:17:31.511042 31323 leader_election.cc:302] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13632200ba98416f81684518fe763abd, 5bbf68dd26db4b209f8aaedcf72f2675; no voters:
I20260808 02:17:31.511231 31711 raft_consensus.cc:2781] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:31.511293 31711 raft_consensus.cc:482] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:31.511327 31711 raft_consensus.cc:3037] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.512022 31711 raft_consensus.cc:504] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.512187 31711 leader_election.cc:288] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [CANDIDATE]: Term 1 election: Requested vote from peers 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281), 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575)
I20260808 02:17:31.512358 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edf709ed8f4411a6559f5451295eb9" candidate_uuid: "13632200ba98416f81684518fe763abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34396a22f4664b5d91623197749a36ac"
I20260808 02:17:31.512442 31604 raft_consensus.cc:3037] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.512950 31212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edf709ed8f4411a6559f5451295eb9" candidate_uuid: "13632200ba98416f81684518fe763abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbf68dd26db4b209f8aaedcf72f2675"
I20260808 02:17:31.513028 31212 raft_consensus.cc:3037] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.513330 31604 raft_consensus.cc:2454] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13632200ba98416f81684518fe763abd in term 1.
I20260808 02:17:31.513561 31321 leader_election.cc:302] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13632200ba98416f81684518fe763abd, 34396a22f4664b5d91623197749a36ac; no voters:
I20260808 02:17:31.513710 31711 raft_consensus.cc:2781] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:31.513754 31212 raft_consensus.cc:2454] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13632200ba98416f81684518fe763abd in term 1.
I20260808 02:17:31.513840 31711 raft_consensus.cc:686] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 1 LEADER]: Becoming Leader. State: Replica: 13632200ba98416f81684518fe763abd, State: Running, Role: LEADER
I20260808 02:17:31.513935 31711 consensus_queue.cc:237] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.514673 30983 catalog_manager.cc:5495] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 13632200ba98416f81684518fe763abd (127.28.177.130). New cstate: current_term: 1 leader_uuid: "13632200ba98416f81684518fe763abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155450, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:31.593978 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.585736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.28.177.190:34935)
0808 02:17:31.586021 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.586024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.586224 (+ 200us) server_negotiation.cc:407] Connection header received
0808 02:17:31.586323 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.586324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.586336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.586385 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.586889 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.587388 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.588111 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.588308 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.588386 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.591008 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.591009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.591010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.591017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.592616 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.593174 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.593175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.593176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.593231 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.593470 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.593471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.593471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.593599 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.593653 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.593842 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.593917 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260808 02:17:31.594107 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.585701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34935 (local address 127.0.0.1:36024)
0808 02:17:31.586134 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.586144 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.586246 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.586399 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.586400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.586740 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:31.586788 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.586791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.587473 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.587474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.587996 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.587997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.588386 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.588387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.588440 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.589218 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:31.589223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:31.590899 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:31.592704 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.592713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.592716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.593091 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.593306 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:31.593307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:31.593307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:31.593382 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:31.593667 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:31.593674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:31.593745 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.593823 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.594015 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":66,"thread_start_us":273,"threads_started":1}
I20260808 02:17:31.596374 30983 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:36024:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:17:31.596680 30983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:31.598912 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 63fc7f41397848e49d40af631faf7808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f1f32c86137245e59785d0542c55409d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:17:31.598991 31389 tablet_service.cc:1467] Processing CreateTablet for tablet 63fc7f41397848e49d40af631faf7808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f1f32c86137245e59785d0542c55409d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:17:31.599057 31579 tablet_service.cc:1467] Processing CreateTablet for tablet 63fc7f41397848e49d40af631faf7808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f1f32c86137245e59785d0542c55409d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:17:31.599179 31579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fc7f41397848e49d40af631faf7808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.599179 31389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fc7f41397848e49d40af631faf7808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.599184 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fc7f41397848e49d40af631faf7808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:31.600858 31702 tablet_bootstrap.cc:492] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: Bootstrap starting.
I20260808 02:17:31.600884 31701 tablet_bootstrap.cc:492] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: Bootstrap starting.
I20260808 02:17:31.601205 31702 tablet_bootstrap.cc:654] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.601261 31701 tablet_bootstrap.cc:654] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.601606 31704 tablet_bootstrap.cc:492] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: Bootstrap starting.
I20260808 02:17:31.601830 31702 tablet_bootstrap.cc:492] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: No bootstrap required, opened a new log
I20260808 02:17:31.601887 31702 ts_tablet_manager.cc:1397] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:31.601979 31702 raft_consensus.cc:348] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.602001 31701 tablet_bootstrap.cc:492] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: No bootstrap required, opened a new log
I20260808 02:17:31.602030 31702 raft_consensus.cc:374] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.602022 31704 tablet_bootstrap.cc:654] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:31.602046 31702 raft_consensus.cc:729] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34396a22f4664b5d91623197749a36ac, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.602048 31701 ts_tablet_manager.cc:1397] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:31.602116 31702 consensus_queue.cc:260] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.602244 31702 ts_tablet_manager.cc:1428] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:31.602214 31701 raft_consensus.cc:348] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.602293 31701 raft_consensus.cc:374] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.602265 31715 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.602319 31701 raft_consensus.cc:729] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbf68dd26db4b209f8aaedcf72f2675, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.602334 31715 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.602355 31701 consensus_queue.cc:260] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.602460 31701 ts_tablet_manager.cc:1428] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:17:31.602583 31709 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.602658 31709 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.603224 31704 tablet_bootstrap.cc:492] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: No bootstrap required, opened a new log
I20260808 02:17:31.603307 31704 ts_tablet_manager.cc:1397] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:17:31.603500 31704 raft_consensus.cc:348] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.603579 31704 raft_consensus.cc:374] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:31.603605 31704 raft_consensus.cc:729] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13632200ba98416f81684518fe763abd, State: Initialized, Role: FOLLOWER
I20260808 02:17:31.603668 31704 consensus_queue.cc:260] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.603797 31704 ts_tablet_manager.cc:1428] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:31.603859 31711 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.603967 31711 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.709635 31709 raft_consensus.cc:482] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:31.709784 31709 raft_consensus.cc:504] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.710181 31709 leader_election.cc:288] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13632200ba98416f81684518fe763abd (127.28.177.130:33235), 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575)
I20260808 02:17:31.713598 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.710492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:59483 (local address 127.28.177.131:44575)
0808 02:17:31.710550 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.710552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.710743 (+ 191us) server_negotiation.cc:407] Connection header received
0808 02:17:31.710868 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.710870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.710883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.710942 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.711128 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.711694 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.712745 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.713191 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.713283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.713295 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.713466 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.713512 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:17:31.713611 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.710571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53281 (local address 127.28.177.130:33235)
0808 02:17:31.710640 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.710642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.710676 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:17:31.710797 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.710798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.710810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.710878 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:31.711139 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.711751 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.712781 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.713208 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.713303 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.713314 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.713500 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.713538 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:17:31.713654 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.710409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44575 (local address 127.28.177.129:59483)
0808 02:17:31.710619 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.710632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.710728 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.710966 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.710966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.710968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.711029 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.711030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.711775 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.711776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.712643 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.712645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.713313 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.713315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.713390 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.713394 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.713438 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.713499 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260808 02:17:31.713660 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.710409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:33235 (local address 127.28.177.129:53281)
0808 02:17:31.710604 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.710618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.710688 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.710898 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.710899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.710902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:31.710999 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.711002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.711849 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.711850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.712653 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.712656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.713323 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.713324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.713399 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.713403 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.713481 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.713519 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":28,"thread_start_us":67,"threads_started":1}
I20260808 02:17:31.713909 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc7f41397848e49d40af631faf7808" candidate_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34396a22f4664b5d91623197749a36ac" is_pre_election: true
I20260808 02:17:31.714004 31604 raft_consensus.cc:2454] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bbf68dd26db4b209f8aaedcf72f2675 in term 0.
I20260808 02:17:31.713985 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc7f41397848e49d40af631faf7808" candidate_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13632200ba98416f81684518fe763abd" is_pre_election: true
I20260808 02:17:31.714093 31418 raft_consensus.cc:2454] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bbf68dd26db4b209f8aaedcf72f2675 in term 0.
I20260808 02:17:31.714188 31110 leader_election.cc:302] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34396a22f4664b5d91623197749a36ac, 5bbf68dd26db4b209f8aaedcf72f2675; no voters:
I20260808 02:17:31.714361 31709 raft_consensus.cc:2781] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:31.714394 31709 raft_consensus.cc:482] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:31.714409 31709 raft_consensus.cc:3037] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.715106 31709 raft_consensus.cc:504] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.715262 31709 leader_election.cc:288] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [CANDIDATE]: Term 1 election: Requested vote from peers 13632200ba98416f81684518fe763abd (127.28.177.130:33235), 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575)
I20260808 02:17:31.715430 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc7f41397848e49d40af631faf7808" candidate_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13632200ba98416f81684518fe763abd"
I20260808 02:17:31.715466 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc7f41397848e49d40af631faf7808" candidate_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34396a22f4664b5d91623197749a36ac"
I20260808 02:17:31.715559 31418 raft_consensus.cc:3037] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.715557 31604 raft_consensus.cc:3037] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:31.716256 31604 raft_consensus.cc:2454] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bbf68dd26db4b209f8aaedcf72f2675 in term 1.
I20260808 02:17:31.716286 31418 raft_consensus.cc:2454] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bbf68dd26db4b209f8aaedcf72f2675 in term 1.
I20260808 02:17:31.716436 31110 leader_election.cc:302] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34396a22f4664b5d91623197749a36ac, 5bbf68dd26db4b209f8aaedcf72f2675; no voters:
I20260808 02:17:31.716585 31709 raft_consensus.cc:2781] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:31.716769 31709 raft_consensus.cc:686] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbf68dd26db4b209f8aaedcf72f2675, State: Running, Role: LEADER
I20260808 02:17:31.716899 31709 consensus_queue.cc:237] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } }
I20260808 02:17:31.717326 31756 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: TxnStatusTablet state changed. Reason: New leader 5bbf68dd26db4b209f8aaedcf72f2675. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.717453 31756 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:17:31.717753 31759 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:17:31.717728 30983 catalog_manager.cc:5495] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:31.717885 31759 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:17:31.752422 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.749334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:34281 (local address 127.0.0.1:41542)
0808 02:17:31.749430 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.749446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.749511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.749763 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.749764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.749765 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:31.749816 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.749818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.750725 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.750726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.751359 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.751361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.751779 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.751781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.751853 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.751868 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.752216 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.752230 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.752286 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.752349 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:17:31.752449 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.749439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.28.177.129:34281)
0808 02:17:31.749520 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.749523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.749537 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:17:31.749619 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.749621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.749655 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.749748 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:31.749891 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.750630 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.751453 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.751701 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.751781 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.751984 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.752110 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.752234 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.752292 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.752314 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:17:31.754071 31604 raft_consensus.cc:1261] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Refusing update from remote peer 5bbf68dd26db4b209f8aaedcf72f2675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:31.754336 31418 raft_consensus.cc:1261] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Refusing update from remote peer 5bbf68dd26db4b209f8aaedcf72f2675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:31.754647 31709 consensus_queue.cc:1035] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:31.754815 31756 consensus_queue.cc:1035] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:31.755604 31715 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: TxnStatusTablet state changed. Reason: New leader 5bbf68dd26db4b209f8aaedcf72f2675. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.755712 31715 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.755645 31711 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: TxnStatusTablet state changed. Reason: New leader 5bbf68dd26db4b209f8aaedcf72f2675. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.755745 31711 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.756552 31709 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.756650 31709 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:17:31.756702 31709 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.756748 31709 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 5bbf68dd26db4b209f8aaedcf72f2675: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:17:31.756803 31720 mvcc.cc:205] Tried to move back new op lower bound from 7316092730381479936 to 7316092730232975360. Current Snapshot: MvccSnapshot[applied={T|T < 7316092730381479936 or (T in {7316092730381479936})}]
I20260808 02:17:31.757093 31726 mvcc.cc:205] Tried to move back new op lower bound from 7316092730381479936 to 7316092730232975360. Current Snapshot: MvccSnapshot[applied={T|T < 7316092730381479936}]
I20260808 02:17:31.757203 31711 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.757282 31711 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 13632200ba98416f81684518fe763abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.758337 31719 mvcc.cc:205] Tried to move back new op lower bound from 7316092730381479936 to 7316092730232975360. Current Snapshot: MvccSnapshot[applied={T|T < 7316092730381479936}]
I20260808 02:17:31.758437 31715 tablet_replica.cc:439] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 } } peers { permanent_uuid: "13632200ba98416f81684518fe763abd" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 33235 } } peers { permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 } } }
I20260808 02:17:31.758561 31715 tablet_replica.cc:442] T 63fc7f41397848e49d40af631faf7808 P 34396a22f4664b5d91623197749a36ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:31.762463 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.759259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.28.177.130:33235)
0808 02:17:31.759302 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.759304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.759313 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:31.759395 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.759396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.759419 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.759511 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:31.759598 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.760332 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.761200 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.761534 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.761610 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.761900 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.762064 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.762149 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.762280 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.762306 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:31.768572 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.765555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:35295 (local address 127.28.177.129:34281)
0808 02:17:31.765666 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.765668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.765678 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:31.765741 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.765742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.765766 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.765811 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:31.765982 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.766477 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.767814 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.768027 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.768104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.768308 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.768351 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.768414 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.768466 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.768491 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:17:31.768600 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.765388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:34281 (local address 127.28.177.130:35295)
0808 02:17:31.765597 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.765611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.765672 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.765823 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.765823 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.765825 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:31.765902 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.765903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.766564 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.766566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.767719 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.767722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.768123 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.768124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.768195 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.768211 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.768415 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.768418 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.768465 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.768511 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260808 02:17:31.771132 31212 raft_consensus.cc:1261] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Refusing update from remote peer 13632200ba98416f81684518fe763abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:31.771132 31604 raft_consensus.cc:1261] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Refusing update from remote peer 13632200ba98416f81684518fe763abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:31.771430 31711 consensus_queue.cc:1035] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bbf68dd26db4b209f8aaedcf72f2675" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:31.771591 31738 consensus_queue.cc:1035] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "34396a22f4664b5d91623197749a36ac" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:31.784077 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.781357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:47243 (local address 127.28.177.130:33235)
0808 02:17:31.781410 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.781413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.781506 (+ 93us) server_negotiation.cc:407] Connection header received
0808 02:17:31.781582 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.781583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.781592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.781669 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:31.781826 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.782308 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.783416 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.783607 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.783660 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.783855 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.783881 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.783916 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.783991 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.784009 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:17:31.784147 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.781271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:33235 (local address 127.28.177.129:47243)
0808 02:17:31.781428 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:31.781441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:31.781513 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:31.781685 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:31.781686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:31.781688 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:31.781763 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.781765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.782377 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.782378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.783325 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:31.783329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.783681 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.783682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:31.783758 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.783774 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:31.783937 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:31.783938 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:17:31.783989 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 02:17:31.784049 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":92,"threads_started":1}
I20260808 02:17:31.792600 30983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:31.792717 30983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:31.793831 30983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 716952982294431bbccaef69c1824b49: Sending DeleteTablet for 3 replicas of tablet 78edf709ed8f4411a6559f5451295eb9
I20260808 02:17:31.794149 31389 tablet_service.cc:1514] Processing DeleteTablet for tablet 78edf709ed8f4411a6559f5451295eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:49934
I20260808 02:17:31.794329 31579 tablet_service.cc:1514] Processing DeleteTablet for tablet 78edf709ed8f4411a6559f5451295eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42430
I20260808 02:17:31.794333 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 78edf709ed8f4411a6559f5451295eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:41526
I20260808 02:17:31.794569 31779 tablet_replica.cc:330] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: stopping tablet replica
I20260808 02:17:31.794667 31779 raft_consensus.cc:2229] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:31.794730 31779 raft_consensus.cc:2258] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:31.795051 31779 ts_tablet_manager.cc:1905] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:31.795292 31777 tablet_replica.cc:330] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: stopping tablet replica
I20260808 02:17:31.795428 31778 tablet_replica.cc:330] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: stopping tablet replica
I20260808 02:17:31.795830 31778 raft_consensus.cc:2229] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:31.795958 31777 raft_consensus.cc:2229] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:17:31.796219 31778 raft_consensus.cc:2258] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:31.796371 31777 raft_consensus.cc:2258] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:31.796706 31778 ts_tablet_manager.cc:1905] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:31.796957 31779 ts_tablet_manager.cc:1918] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:31.797086 31779 log.cc:1196] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/78edf709ed8f4411a6559f5451295eb9
I20260808 02:17:31.797106 31777 ts_tablet_manager.cc:1905] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:31.797464 31779 ts_tablet_manager.cc:1939] T 78edf709ed8f4411a6559f5451295eb9 P 5bbf68dd26db4b209f8aaedcf72f2675: Deleting consensus metadata
I20260808 02:17:31.797953 30965 catalog_manager.cc:4841] TS 5bbf68dd26db4b209f8aaedcf72f2675 (127.28.177.129:34281): tablet 78edf709ed8f4411a6559f5451295eb9 (table test-table [id=16647ce32a9945c485a0f08f02d7c160]) successfully deleted
I20260808 02:17:31.798942 31778 ts_tablet_manager.cc:1918] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:31.799019 31778 log.cc:1196] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/78edf709ed8f4411a6559f5451295eb9
I20260808 02:17:31.799247 31778 ts_tablet_manager.cc:1939] T 78edf709ed8f4411a6559f5451295eb9 P 34396a22f4664b5d91623197749a36ac: Deleting consensus metadata
I20260808 02:17:31.799620 30963 catalog_manager.cc:4841] TS 34396a22f4664b5d91623197749a36ac (127.28.177.131:44575): tablet 78edf709ed8f4411a6559f5451295eb9 (table test-table [id=16647ce32a9945c485a0f08f02d7c160]) successfully deleted
I20260808 02:17:31.801057 31777 ts_tablet_manager.cc:1918] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:31.801141 31777 log.cc:1196] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/78edf709ed8f4411a6559f5451295eb9
I20260808 02:17:31.801367 31777 ts_tablet_manager.cc:1939] T 78edf709ed8f4411a6559f5451295eb9 P 13632200ba98416f81684518fe763abd: Deleting consensus metadata
I20260808 02:17:31.801846 30964 catalog_manager.cc:4841] TS 13632200ba98416f81684518fe763abd (127.28.177.130:33235): tablet 78edf709ed8f4411a6559f5451295eb9 (table test-table [id=16647ce32a9945c485a0f08f02d7c160]) successfully deleted
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[30921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:31.813445 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.795738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.28.177.129:34281)
0808 02:17:31.795782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.795784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.801633 (+ 5849us) server_negotiation.cc:407] Connection header received
0808 02:17:31.801834 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.801836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.801848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.801882 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.802693 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.803459 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.804652 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.804998 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.805047 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.809271 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.809281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.809282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.809291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.811855 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.812332 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.812336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.812337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.812407 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.812611 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.812612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.812612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.812785 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.812988 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.813224 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.813336 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 02:17:31.814168 31267 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41548
I20260808 02:17:31.823419 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:31.815228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.28.177.190:34935)
0808 02:17:31.815277 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:31.815279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:31.815516 (+ 237us) server_negotiation.cc:407] Connection header received
0808 02:17:31.815561 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:31.815562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:31.815576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:31.815646 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:31.816313 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.817144 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.818239 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:31.818540 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:31.818636 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:31.820142 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:31.820143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:31.820144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:31.820153 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:31.821991 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.822444 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.822445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.822446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.822515 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:31.822819 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:31.822820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:31.822820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:31.822974 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:31.823084 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:31.823233 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:17:31.823321 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 02:17:31.823664 30983 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36034
I20260808 02:17:31.824440 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 31091
I20260808 02:17:31.848286 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 31296
I20260808 02:17:31.859416 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 31487
I20260808 02:17:31.870826 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 30940
2026-08-08T02:17:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2806 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[31786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[31786](info): set up 2 sockets
Aug 08 02:17:31 dist-test-slave-3h5h krb5kdc[31786](info): commencing operation
krb5kdc: starting...
W20260808 02:17:33.199666 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:33.337198 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:45947
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:33.426326 31800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:33.426563 31800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:33.426589 31800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:33.427883 31800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:33.427927 31800 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:17:33.427940 31800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:33.427955 31800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:33.427970 31800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:33.427982 31800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:33.429607 31800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:45947
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:33.429888 31800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:33.430150 31800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:33.435336 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.435339 31814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.435436 31811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:33.436360 31800 server_base.cc:1034] running on GCE node
I20260808 02:17:33.436697 31800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:33.437072 31800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:33.438253 31800 hybrid_clock.cc:648] HybridClock initialized: now 1786155453438229 us; error 38 us; skew 500 ppm
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:33.441022 31800 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:33.441465 31800 webserver.cc:460] Webserver started at http://127.28.177.190:40905/ using document root <none> and password file <none>
I20260808 02:17:33.441666 31800 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:33.441707 31800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:33.441805 31800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:33.442896 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "a783be595e604e278579ca746a56f5c3"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "8ccf7bf4b3cf549591c43222161ec7e0"
server_key_iv: "9cd299c8e39ba18b80ba2b8c316b9c8e"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.443261 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "a783be595e604e278579ca746a56f5c3"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "8ccf7bf4b3cf549591c43222161ec7e0"
server_key_iv: "9cd299c8e39ba18b80ba2b8c316b9c8e"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.444568 31800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:33.445391 31823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.445606 31800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:33.445720 31800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "a783be595e604e278579ca746a56f5c3"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "8ccf7bf4b3cf549591c43222161ec7e0"
server_key_iv: "9cd299c8e39ba18b80ba2b8c316b9c8e"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.445794 31800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:33.454883 31800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:33.457201 31800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:33.457368 31800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:33.462250 31800 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:45947
I20260808 02:17:33.462338 31916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:45947 every 8 connection(s)
I20260808 02:17:33.462693 31800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:33.463294 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 31800
I20260808 02:17:33.463402 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:33.463528 31917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:33.463598 29382 external_mini_cluster.cc:1428] Setting key a6e551de99e57ebfbbee18083c34edca
I20260808 02:17:33.467526 31917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Bootstrap starting.
I20260808 02:17:33.468407 31917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:33.468791 31917 log.cc:824] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:33.469760 31917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: No bootstrap required, opened a new log
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:33.472942 31917 raft_consensus.cc:348] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a783be595e604e278579ca746a56f5c3" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45947 } }
I20260808 02:17:33.473206 31917 raft_consensus.cc:374] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:33.473282 31917 raft_consensus.cc:729] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a783be595e604e278579ca746a56f5c3, State: Initialized, Role: FOLLOWER
I20260808 02:17:33.473466 31917 consensus_queue.cc:260] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a783be595e604e278579ca746a56f5c3" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45947 } }
I20260808 02:17:33.473596 31917 raft_consensus.cc:388] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:33.473695 31917 raft_consensus.cc:482] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:33.473764 31917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:33.474696 31917 raft_consensus.cc:504] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a783be595e604e278579ca746a56f5c3" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45947 } }
I20260808 02:17:33.474848 31917 leader_election.cc:302] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a783be595e604e278579ca746a56f5c3; no voters:
I20260808 02:17:33.475097 31917 leader_election.cc:288] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:33.475198 31923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:33.475407 31923 raft_consensus.cc:686] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [term 1 LEADER]: Becoming Leader. State: Replica: a783be595e604e278579ca746a56f5c3, State: Running, Role: LEADER
I20260808 02:17:33.475425 31917 sys_catalog.cc:564] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:33.475549 31923 consensus_queue.cc:237] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a783be595e604e278579ca746a56f5c3" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45947 } }
I20260808 02:17:33.475931 31923 sys_catalog.cc:455] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a783be595e604e278579ca746a56f5c3. Latest consensus state: current_term: 1 leader_uuid: "a783be595e604e278579ca746a56f5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a783be595e604e278579ca746a56f5c3" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45947 } } }
I20260808 02:17:33.476148 31923 sys_catalog.cc:458] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:33.476378 31935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:33.475945 31924 sys_catalog.cc:455] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a783be595e604e278579ca746a56f5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a783be595e604e278579ca746a56f5c3" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45947 } } }
I20260808 02:17:33.476729 31924 sys_catalog.cc:458] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:33.477406 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.464163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.28.177.190:45947)
0808 02:17:33.464345 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:33.464353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:33.464379 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:17:33.465063 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:33.465071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:33.465122 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:33.465241 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:33.466394 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.467340 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.468604 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.469304 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.469491 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.472504 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:33.472511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:33.472511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:33.472529 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:33.475029 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.475567 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.475571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.475571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.475684 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.475934 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.475935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.475935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.476403 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:33.476589 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:33.476985 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 02:17:33.477150 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260808 02:17:33.477680 31935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:33.479555 31935 catalog_manager.cc:1347] Generated new cluster ID: 40960fc71e1247709bbb51fcce7d8b93
I20260808 02:17:33.479611 31935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:33.492525 31935 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:33.493167 31935 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:33.498744 31935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Generated new TSK 0
I20260808 02:17:33.499060 31935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:33.535754 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45947
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:33.629359 31953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:33.629616 31953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:33.629669 31953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:33.630906 31953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:33.630981 31953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:33.631017 31953 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:17:33.631047 31953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:17:33.631081 31953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:33.632737 31953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:33.633055 31953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:33.633301 31953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:33.638979 31960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.638968 31962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.639221 31959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:33.639256 31953 server_base.cc:1034] running on GCE node
I20260808 02:17:33.639478 31953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:33.639744 31953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:33.640918 31953 hybrid_clock.cc:648] HybridClock initialized: now 1786155453640889 us; error 42 us; skew 500 ppm
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:33.643767 31953 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:33.644274 31953 webserver.cc:460] Webserver started at http://127.28.177.129:36157/ using document root <none> and password file <none>
I20260808 02:17:33.644537 31953 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:33.644614 31953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:33.644784 31953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:33.646011 31953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "3a71157043bf4d4ea60661df405aaf02"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "d9f9f0e976cca0b646c8e33a42878041"
server_key_iv: "cd54e63e11b5ce484f6e4d82d3fc4eed"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.646426 31953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "3a71157043bf4d4ea60661df405aaf02"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "d9f9f0e976cca0b646c8e33a42878041"
server_key_iv: "cd54e63e11b5ce484f6e4d82d3fc4eed"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.647945 31953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:33.649019 31971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.649338 31953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:33.649482 31953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "3a71157043bf4d4ea60661df405aaf02"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "d9f9f0e976cca0b646c8e33a42878041"
server_key_iv: "cd54e63e11b5ce484f6e4d82d3fc4eed"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.649595 31953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:33.660526 31953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:33.662914 31953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:33.663038 31953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:33.663556 31953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:33.663594 31953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.663618 31953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:33.663631 31953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.670884 31953 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45391
I20260808 02:17:33.670986 32134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45391 every 8 connection(s)
I20260808 02:17:33.671355 31953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:33.672458 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 31953
I20260808 02:17:33.672569 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:33.672813 29382 external_mini_cluster.cc:1428] Setting key f3d3dac35ce68a9c6ce2c91068adaa6b
I20260808 02:17:33.676688 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.664267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:49369 (local address 127.28.177.190:45947)
0808 02:17:33.664333 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:33.664336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:33.664579 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:17:33.665393 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:33.665395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:33.665411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:33.665488 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:33.666485 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.667339 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.669091 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.669462 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.669576 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.672463 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:33.672465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:33.672465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:33.672475 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:33.674685 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.675184 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.675185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.675186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.675260 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.675723 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.675725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.675725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.675870 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:33.676010 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:33.676430 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 02:17:33.676589 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:17:33.676901 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.664251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.28.177.129:49369)
0808 02:17:33.664438 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:33.664480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:33.665190 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:33.665593 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:33.665599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:33.666039 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:33.666365 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.666372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.667454 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.667455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.668981 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.668984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.669588 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.669589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.669810 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.670479 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:33.670489 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:33.672354 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:33.674782 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.674790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.674791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.675097 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.675428 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.675430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.675430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.675627 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.675975 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:33.675983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:33.676251 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 02:17:33.676404 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 02:17:33.676605 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260808 02:17:33.677625 32135 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45947
I20260808 02:17:33.677721 32135 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:33.677940 32135 heartbeater.cc:511] Master 127.28.177.190:45947 requested a full tablet report, sending...
I20260808 02:17:33.678587 31854 ts_manager.cc:194] Registered new tserver with Master: 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391)
I20260808 02:17:33.679562 31854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:49369
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:17:33.685912 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.678909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.28.177.129:39303)
0808 02:17:33.679077 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:33.679090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:33.679147 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:33.679307 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:33.679308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:33.679528 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:33.679602 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.679604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.680619 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.680620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.681245 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.681247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.681766 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.681767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.681836 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.682464 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:33.682468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:33.682779 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:33.684595 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.684596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.684597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.684804 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.685227 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.685228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.685229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.685294 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.685631 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:33.685632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:33.685675 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 02:17:33.685743 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:17:33.685836 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260808 02:17:33.685947 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.679111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:39303 (local address 127.28.177.190:45947)
0808 02:17:33.679153 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:33.679156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:33.679169 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:17:33.679196 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:33.679197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:33.679208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:33.679288 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:33.679709 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.680495 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.681332 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.681675 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.681759 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.682879 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:33.682881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:33.682882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:33.682891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:33.684489 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.684968 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.684969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.685002 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.685070 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.685416 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.685417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.685418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.685548 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:33.685633 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:33.685804 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:17:33.685880 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:33.704711 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45947
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:33.794201 32147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:33.794427 32147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:33.794454 32147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:33.795651 32147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:33.795699 32147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:33.795722 32147 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:17:33.795787 32147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:17:33.795814 32147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:33.797376 32147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:33.797646 32147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:33.797895 32147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:33.803511 32155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.803560 32156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.803524 32159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:33.804050 32147 server_base.cc:1034] running on GCE node
I20260808 02:17:33.804283 32147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:33.804612 32147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:33.805816 32147 hybrid_clock.cc:648] HybridClock initialized: now 1786155453805776 us; error 57 us; skew 500 ppm
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:33.808719 32147 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:33.809357 32147 webserver.cc:460] Webserver started at http://127.28.177.130:45507/ using document root <none> and password file <none>
I20260808 02:17:33.809648 32147 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:33.809729 32147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:33.809906 32147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:33.811113 32147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "9fb40e9366054c12949cd4278bbc7f3f"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "c678d0bcc5ff65187a5bdbffecafbaa1"
server_key_iv: "cf4239989015cb032ba191a1ee3001c9"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.811517 32147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb40e9366054c12949cd4278bbc7f3f"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "c678d0bcc5ff65187a5bdbffecafbaa1"
server_key_iv: "cf4239989015cb032ba191a1ee3001c9"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.813419 32147 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:17:33.814416 32167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.814677 32147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:33.814795 32147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "9fb40e9366054c12949cd4278bbc7f3f"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "c678d0bcc5ff65187a5bdbffecafbaa1"
server_key_iv: "cf4239989015cb032ba191a1ee3001c9"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.814903 32147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:33.823731 32147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:33.826170 32147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:33.826347 32147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:33.826949 32147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:33.827010 32147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.827046 32147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:33.827067 32147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:33.834895 32147 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:44429
I20260808 02:17:33.835042 32336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:44429 every 8 connection(s)
I20260808 02:17:33.835340 32147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:17:33.837479 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.827482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:32971 (local address 127.28.177.190:45947)
0808 02:17:33.827607 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:33.827609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:33.827818 (+ 209us) server_negotiation.cc:407] Connection header received
0808 02:17:33.828582 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:33.828583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:33.828598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:33.828675 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:33.829641 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.830215 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.831376 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.831582 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.831665 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.834131 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:33.834132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:33.834133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:33.834141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:33.835857 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.836405 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.836407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.836407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.836474 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.836747 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.836748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.836748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.836933 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:33.837010 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:33.837292 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 02:17:33.837385 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:17:33.837678 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.827453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.28.177.130:32971)
0808 02:17:33.827654 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:33.827712 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:33.828455 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:33.828756 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:33.828761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:33.829213 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:33.829530 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.829537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.830326 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.830327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.831257 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.831260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.831686 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.831687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.831845 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.832464 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:33.832490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:33.834010 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:33.835953 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.835981 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.835984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.836313 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.836542 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.836543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.836543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.836667 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.837001 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:33.837009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:33.837154 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 02:17:33.837270 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 02:17:33.837437 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260808 02:17:33.838270 32337 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45947
I20260808 02:17:33.838351 32337 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:33.838567 32337 heartbeater.cc:511] Master 127.28.177.190:45947 requested a full tablet report, sending...
I20260808 02:17:33.838979 31854 ts_manager.cc:194] Registered new tserver with Master: 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429)
I20260808 02:17:33.839344 31854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:32971
I20260808 02:17:33.841084 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 32147
I20260808 02:17:33.841180 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:33.841364 29382 external_mini_cluster.cc:1428] Setting key ec52fa96efd54f325071f1d5c685908b
I20260808 02:17:33.845803 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.839435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:52359 (local address 127.28.177.190:45947)
0808 02:17:33.839477 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:33.839479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:33.839623 (+ 144us) server_negotiation.cc:407] Connection header received
0808 02:17:33.839704 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:33.839705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:33.839715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:33.839764 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:33.840101 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.840576 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.841608 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.841855 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.841962 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.843015 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:33.843016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:33.843017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:33.843023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:33.844491 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.844912 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.844914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.844914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.844969 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:33.845261 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:33.845263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:33.845263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:33.845420 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:33.845494 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:33.845667 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:17:33.845738 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:17:33.845932 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.839364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.28.177.130:52359)
0808 02:17:33.839536 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:33.839546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:33.839638 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:33.839767 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:33.839768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:33.839947 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:33.840015 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.840016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.840673 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.840675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.841487 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.841490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.841964 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.841965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.842048 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:33.842483 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:33.842489 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:33.842923 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:33.844581 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.844583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.844583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.844814 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.845042 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:33.845043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:33.845043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:33.845124 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:33.845504 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:33.845505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:33.845558 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 02:17:33.845635 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:17:33.845765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:33.873234 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45947
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:33.961117 32347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:33.961359 32347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:33.961402 32347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:33.962610 32347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:33.962679 32347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:33.962715 32347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:17:33.962755 32347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:17:33.962793 32347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:33.964318 32347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:33.964607 32347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:33.964913 32347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:33.970533 32352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.970520 32353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:33.970520 32355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:33.970800 32347 server_base.cc:1034] running on GCE node
I20260808 02:17:33.971138 32347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:33.971378 32347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:33.972534 32347 hybrid_clock.cc:648] HybridClock initialized: now 1786155453972541 us; error 53 us; skew 500 ppm
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:33.975795 32347 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:33.976506 32347 webserver.cc:460] Webserver started at http://127.28.177.131:33661/ using document root <none> and password file <none>
I20260808 02:17:33.976787 32347 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:33.976886 32347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:33.977082 32347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:33.978255 32347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "17b1cea48a24457796fc2d77733195e1"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "14a8c8a649feea753bc939cc4d2209e5"
server_key_iv: "92b34db1a43f3bec899be5497fe1a01a"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.978639 32347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "17b1cea48a24457796fc2d77733195e1"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "14a8c8a649feea753bc939cc4d2209e5"
server_key_iv: "92b34db1a43f3bec899be5497fe1a01a"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.980266 32347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:33.981467 32364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.981690 32347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:33.981809 32347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "17b1cea48a24457796fc2d77733195e1"
format_stamp: "Formatted at 2026-08-08 02:17:33 on dist-test-slave-3h5h"
server_key: "14a8c8a649feea753bc939cc4d2209e5"
server_key_iv: "92b34db1a43f3bec899be5497fe1a01a"
server_key_version: "encryptionkey@0"
I20260808 02:17:33.981902 32347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:33.991139 32347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:33.993674 32347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:33.993885 32347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:33.994541 32347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:33.994606 32347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:33.994652 32347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:33.994683 32347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:17:33 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:34.002538 32347 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:44015
I20260808 02:17:34.002898 32347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:17:34.003371 32524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:44015 every 8 connection(s)
I20260808 02:17:34.006114 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.995054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:57919 (local address 127.28.177.190:45947)
0808 02:17:33.995125 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:33.995128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:33.995342 (+ 214us) server_negotiation.cc:407] Connection header received
0808 02:17:33.995860 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:33.995862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:33.995877 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:33.995957 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:33.996753 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.997377 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.998834 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.999109 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.999219 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.003049 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:34.003051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:34.003052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:34.003062 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:34.004642 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.005259 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.005260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.005261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.005320 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.005570 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.005570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.005571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.005680 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:34.005744 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.005954 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.006026 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:17:34.006459 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:33.994979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.28.177.131:57919)
0808 02:17:33.995169 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:33.995218 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:33.995754 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:33.996145 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:33.996149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:33.996459 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:33.996650 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.996655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.997503 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.997504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.998706 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:33.998709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:33.999229 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:33.999231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:33.999404 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.000082 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:34.000094 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:34.002890 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:34.004731 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:34.004738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:34.004739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:34.005175 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:34.005390 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:34.005390 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:34.005391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:34.005508 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:34.005723 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:34.005726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:34.005844 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.005939 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.006144 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260808 02:17:34.006991 32525 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45947
I20260808 02:17:34.007090 32525 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:34.007304 32525 heartbeater.cc:511] Master 127.28.177.190:45947 requested a full tablet report, sending...
I20260808 02:17:34.007730 31853 ts_manager.cc:194] Registered new tserver with Master: 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:34.008128 31853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:57919
I20260808 02:17:34.010151 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 32347
I20260808 02:17:34.010262 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:34.010484 29382 external_mini_cluster.cc:1428] Setting key 3e82e28c63d4c05f11e313e6670823cf
I20260808 02:17:34.011475 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:17:34.014792 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.008324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:49557 (local address 127.28.177.190:45947)
0808 02:17:34.008365 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.008367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.008505 (+ 138us) server_negotiation.cc:407] Connection header received
0808 02:17:34.008540 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.008540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.008549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.008596 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:34.009055 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.009626 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.010410 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.010798 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.010887 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.011909 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:34.011911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:34.011911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:34.011920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:34.013461 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.013922 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.013923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.013923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.013990 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.014220 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.014221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.014221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.014344 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:34.014432 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.014613 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.014691 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:17:34.014812 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.008272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.28.177.131:49557)
0808 02:17:34.008400 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.008410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.008499 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.008641 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.008642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.008880 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:34.008957 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.008959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.009735 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.009736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.010271 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.010274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.010888 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.010889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.010974 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.011439 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:34.011443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:34.011815 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:34.013622 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:34.013623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:34.013624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:34.013827 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:34.014070 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:34.014071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:34.014072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:34.014140 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:34.014421 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:34.014422 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:34.014473 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.014567 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.014688 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
Aug 08 02:17:34 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:17:34 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:34.039748 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.031272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.28.177.190:45947)
0808 02:17:34.031347 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.031349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.031360 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:34.031416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.031417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.031436 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.031511 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:34.032185 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.032764 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.033642 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.033871 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.033961 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.036715 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:34.036717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:34.036718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:34.036727 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:34.038334 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.038732 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.038733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.038734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.038807 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.039092 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.039093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.039094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.039228 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:34.039356 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.039537 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.039628 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:17:34.041723 31853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:34.043411 31853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:34.050209 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.045970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44429 (local address 127.0.0.1:42038)
0808 02:17:34.046189 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.046231 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.046302 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.046689 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.046692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.046695 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.046778 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.046780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.047889 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.047890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.049017 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.049019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.049854 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.049855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.049939 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.049952 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.050061 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.050119 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260808 02:17:34.050221 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.046280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.28.177.130:44429)
0808 02:17:34.046489 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.046494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.046505 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:34.046550 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.046553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.046570 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.046674 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.046883 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.047784 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.049114 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.049757 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.049845 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.049906 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.050082 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.050120 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260808 02:17:34.050979 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.045970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44015 (local address 127.0.0.1:50920)
0808 02:17:34.046634 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.046646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.046703 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.047238 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.047239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.047240 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.047295 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.047296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.048367 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.048368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.049591 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.049594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.050690 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.050692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.050770 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.050776 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.050836 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.050887 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":7,"thread_start_us":101,"threads_started":1}
I20260808 02:17:34.051348 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.046266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.28.177.131:44015)
0808 02:17:34.046473 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.046478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.046771 (+ 293us) server_negotiation.cc:407] Connection header received
0808 02:17:34.046821 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.046824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.046840 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.046919 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.047384 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.048274 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.049699 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.050509 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.050596 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.050651 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.051175 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.051230 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260808 02:17:34.051297 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 186df0d418d342298007769d72acc0df (DEFAULT_TABLE table=test-table [id=81f484c7ddb840699e82e964b639bce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:34.051749 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186df0d418d342298007769d72acc0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.052146 32438 tablet_service.cc:1467] Processing CreateTablet for tablet 186df0d418d342298007769d72acc0df (DEFAULT_TABLE table=test-table [id=81f484c7ddb840699e82e964b639bce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:34.052490 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186df0d418d342298007769d72acc0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.055280 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.045974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45391 (local address 127.0.0.1:56450)
0808 02:17:34.046368 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.046378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.046427 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.047213 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.047214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.047216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.047285 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.047287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.052661 (+ 5374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.052662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.053879 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.053882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.054966 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.054967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.055064 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.055069 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.055137 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.055179 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":100,"threads_started":1}
I20260808 02:17:34.055352 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.046172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.28.177.129:45391)
0808 02:17:34.047057 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.047061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.047070 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:34.047102 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.047104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.047116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.047181 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.048409 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.052563 (+ 4154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.053982 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.054855 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.054946 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.055004 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.055172 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.055219 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":99,"threads_started":1}
I20260808 02:17:34.055995 32558 tablet_bootstrap.cc:492] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Bootstrap starting.
I20260808 02:17:34.056066 32560 tablet_bootstrap.cc:492] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Bootstrap starting.
I20260808 02:17:34.056382 32044 tablet_service.cc:1467] Processing CreateTablet for tablet 186df0d418d342298007769d72acc0df (DEFAULT_TABLE table=test-table [id=81f484c7ddb840699e82e964b639bce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:34.056659 32558 tablet_bootstrap.cc:654] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.056818 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186df0d418d342298007769d72acc0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.057063 32558 log.cc:824] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:34.057081 32560 tablet_bootstrap.cc:654] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.057461 32560 log.cc:824] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:34.057961 32558 tablet_bootstrap.cc:492] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: No bootstrap required, opened a new log
I20260808 02:17:34.058035 32558 ts_tablet_manager.cc:1397] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:17:34.058331 32560 tablet_bootstrap.cc:492] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: No bootstrap required, opened a new log
I20260808 02:17:34.058415 32560 ts_tablet_manager.cc:1397] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:34.059811 32558 raft_consensus.cc:348] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.059973 32558 raft_consensus.cc:374] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.060019 32558 raft_consensus.cc:729] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb40e9366054c12949cd4278bbc7f3f, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.060160 32558 consensus_queue.cc:260] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.060468 32337 heartbeater.cc:503] Master 127.28.177.190:45947 was elected leader, sending a full tablet report...
I20260808 02:17:34.060746 32558 ts_tablet_manager.cc:1428] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:17:34.060772 32560 raft_consensus.cc:348] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.061050 32560 raft_consensus.cc:374] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.061116 32560 raft_consensus.cc:729] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b1cea48a24457796fc2d77733195e1, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.061239 32564 tablet_bootstrap.cc:492] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Bootstrap starting.
I20260808 02:17:34.061239 32560 consensus_queue.cc:260] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.061534 32560 ts_tablet_manager.cc:1428] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:34.061617 32525 heartbeater.cc:503] Master 127.28.177.190:45947 was elected leader, sending a full tablet report...
I20260808 02:17:34.062156 32564 tablet_bootstrap.cc:654] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.062534 32564 log.cc:824] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:34.063381 32564 tablet_bootstrap.cc:492] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: No bootstrap required, opened a new log
I20260808 02:17:34.063470 32564 ts_tablet_manager.cc:1397] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:17:34.064817 32564 raft_consensus.cc:348] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.065033 32564 raft_consensus.cc:374] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.065089 32564 raft_consensus.cc:729] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.065203 32564 consensus_queue.cc:260] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.065483 32564 ts_tablet_manager.cc:1428] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:17:34.065562 32135 heartbeater.cc:503] Master 127.28.177.190:45947 was elected leader, sending a full tablet report...
W20260808 02:17:34.086097 32338 tablet.cc:2369] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:34.123659 32568 raft_consensus.cc:482] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:34.123811 32568 raft_consensus.cc:504] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.124279 32568 leader_election.cc:288] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015), 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429)
I20260808 02:17:34.129004 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.124735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:37933 (local address 127.28.177.131:44015)
0808 02:17:34.124783 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.124785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.124885 (+ 100us) server_negotiation.cc:407] Connection header received
0808 02:17:34.124992 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.124993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.125002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.125065 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.125245 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.125993 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.127795 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.128523 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.128616 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.128627 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.128871 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.128919 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:17:34.129024 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.124503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44429 (local address 127.28.177.129:51823)
0808 02:17:34.124560 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.124573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.124622 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.125008 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.125009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.125010 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.125112 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.125115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.126103 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.126105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.127656 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.127659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.128634 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.128635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.128739 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.128747 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.128816 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.128900 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":145}
I20260808 02:17:34.129037 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.124507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44015 (local address 127.28.177.129:37933)
0808 02:17:34.124772 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.124785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.124902 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.125093 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.125094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.125095 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.125165 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.125167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.126103 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.126105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.127659 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.127661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.128634 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.128635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.128739 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.128747 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.128816 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.128900 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":182,"thread_start_us":133,"threads_started":1}
I20260808 02:17:34.129114 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.124735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:51823 (local address 127.28.177.130:44429)
0808 02:17:34.124791 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.124793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.124801 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:34.124887 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.124888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.124900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.124991 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.125199 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.125968 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.127794 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.128512 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.128616 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.128627 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.128871 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.128921 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:17:34.129454 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186df0d418d342298007769d72acc0df" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f" is_pre_election: true
I20260808 02:17:34.129488 32463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186df0d418d342298007769d72acc0df" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1" is_pre_election: true
I20260808 02:17:34.129660 32270 raft_consensus.cc:2454] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 0.
I20260808 02:17:34.129660 32463 raft_consensus.cc:2454] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 0.
I20260808 02:17:34.129956 31974 leader_election.cc:302] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b1cea48a24457796fc2d77733195e1, 3a71157043bf4d4ea60661df405aaf02; no voters:
I20260808 02:17:34.130162 32568 raft_consensus.cc:2781] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:34.130223 32568 raft_consensus.cc:482] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:34.130240 32568 raft_consensus.cc:3037] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.131129 32568 raft_consensus.cc:504] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.131289 32568 leader_election.cc:288] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 election: Requested vote from peers 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015), 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429)
I20260808 02:17:34.131470 32463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186df0d418d342298007769d72acc0df" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1"
I20260808 02:17:34.131482 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186df0d418d342298007769d72acc0df" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f"
I20260808 02:17:34.131573 32270 raft_consensus.cc:3037] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.131573 32463 raft_consensus.cc:3037] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.132598 32270 raft_consensus.cc:2454] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 1.
I20260808 02:17:34.132598 32463 raft_consensus.cc:2454] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 1.
I20260808 02:17:34.132800 31974 leader_election.cc:302] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b1cea48a24457796fc2d77733195e1, 3a71157043bf4d4ea60661df405aaf02; no voters:
I20260808 02:17:34.132927 32568 raft_consensus.cc:2781] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:34.133159 32568 raft_consensus.cc:686] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 1 LEADER]: Becoming Leader. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Running, Role: LEADER
I20260808 02:17:34.133250 32568 consensus_queue.cc:237] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } }
I20260808 02:17:34.134654 31853 catalog_manager.cc:5495] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "3a71157043bf4d4ea60661df405aaf02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } health_report { overall_health: UNKNOWN } } }
W20260808 02:17:34.172286 32136 tablet.cc:2369] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:17:34 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155453, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:34.177721 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.169633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45947 (local address 127.0.0.1:52866)
0808 02:17:34.169823 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.169834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.169904 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.170059 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.170060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.170370 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:34.170421 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.170423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.171134 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.171135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.171737 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.171739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.172155 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.172156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.172211 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.172803 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:34.172808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:34.174420 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:34.176221 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:34.176228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:34.176231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:34.176676 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:34.176965 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:34.176966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:34.176966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:34.177034 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:34.177342 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:34.177356 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:34.177405 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.177468 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.177611 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260808 02:17:34.177772 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.169722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.28.177.190:45947)
0808 02:17:34.169816 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.169820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.169900 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:17:34.169949 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.169950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.169959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.170003 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:34.170501 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.171021 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.171824 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.172007 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.172075 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.174518 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:34.174520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:34.174520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:34.174527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:34.176129 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.176765 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.176767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.176767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.176886 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:34.177105 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:34.177106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:34.177106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:34.177267 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:34.177325 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.177522 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.177672 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":65}
I20260808 02:17:34.180104 31853 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:52866:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:17:34.180297 31853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:34.181978 32438 tablet_service.cc:1467] Processing CreateTablet for tablet a1cae8a8f9564f92baf7242122b0cf03 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=719870cda61840f4ab9c4d0b03c5d473]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:17:34.182082 32044 tablet_service.cc:1467] Processing CreateTablet for tablet a1cae8a8f9564f92baf7242122b0cf03 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=719870cda61840f4ab9c4d0b03c5d473]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:17:34.182197 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cae8a8f9564f92baf7242122b0cf03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.182230 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cae8a8f9564f92baf7242122b0cf03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.183760 32564 tablet_bootstrap.cc:492] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: Bootstrap starting.
I20260808 02:17:34.183835 32560 tablet_bootstrap.cc:492] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: Bootstrap starting.
I20260808 02:17:34.184247 32564 tablet_bootstrap.cc:654] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.184419 32560 tablet_bootstrap.cc:654] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.185024 32564 tablet_bootstrap.cc:492] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: No bootstrap required, opened a new log
I20260808 02:17:34.185014 32243 tablet_service.cc:1467] Processing CreateTablet for tablet a1cae8a8f9564f92baf7242122b0cf03 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=719870cda61840f4ab9c4d0b03c5d473]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:17:34.185111 32564 ts_tablet_manager.cc:1397] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:34.185228 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cae8a8f9564f92baf7242122b0cf03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.185279 32560 tablet_bootstrap.cc:492] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: No bootstrap required, opened a new log
I20260808 02:17:34.185346 32560 ts_tablet_manager.cc:1397] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:17:34.185282 32564 raft_consensus.cc:348] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.185402 32564 raft_consensus.cc:374] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.185434 32564 raft_consensus.cc:729] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.185500 32564 consensus_queue.cc:260] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.185504 32560 raft_consensus.cc:348] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.185583 32560 raft_consensus.cc:374] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.185611 32560 raft_consensus.cc:729] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b1cea48a24457796fc2d77733195e1, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.185621 32564 ts_tablet_manager.cc:1428] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:34.185653 32568 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.185675 32560 consensus_queue.cc:260] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.185739 32568 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:34.185788 32560 ts_tablet_manager.cc:1428] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:34.185985 32565 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.186076 32565 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:34.186759 32558 tablet_bootstrap.cc:492] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: Bootstrap starting.
I20260808 02:17:34.187566 32558 tablet_bootstrap.cc:654] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.188297 32558 tablet_bootstrap.cc:492] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: No bootstrap required, opened a new log
I20260808 02:17:34.188386 32558 ts_tablet_manager.cc:1397] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:17:34.188596 32558 raft_consensus.cc:348] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.188730 32558 raft_consensus.cc:374] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.188756 32558 raft_consensus.cc:729] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb40e9366054c12949cd4278bbc7f3f, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.188853 32558 consensus_queue.cc:260] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.189002 32558 ts_tablet_manager.cc:1428] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:34.189159 32563 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.189239 32563 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 02:17:34.253895 32526 tablet.cc:2369] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:34.301045 32563 raft_consensus.cc:482] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:34.301167 32563 raft_consensus.cc:504] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.301597 32563 leader_election.cc:288] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391), 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:34.305022 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.301804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44015 (local address 127.28.177.130:46287)
0808 02:17:34.302269 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.302282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.302346 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.302493 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.302494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.302496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.302579 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.302581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.303206 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.303207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.304124 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.304127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.304706 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.304708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.304786 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.304789 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.304888 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.304930 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":218,"thread_start_us":139,"threads_started":1}
I20260808 02:17:34.305006 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.301916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:46287 (local address 127.28.177.131:44015)
0808 02:17:34.301997 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.302000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.302336 (+ 336us) server_negotiation.cc:407] Connection header received
0808 02:17:34.302413 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.302413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.302421 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.302486 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.302654 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.303116 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.304212 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.304587 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.304676 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.304685 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.304900 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.304938 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:17:34.305308 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.301929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:35861 (local address 127.28.177.129:45391)
0808 02:17:34.301997 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.302000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.302011 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:34.302188 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.302190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.302201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.302263 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.302470 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.303061 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.304244 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.304675 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.304776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.304785 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.305209 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.305262 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:34.305300 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.301792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45391 (local address 127.28.177.130:35861)
0808 02:17:34.301869 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.301885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.302058 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.302299 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.302300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.302302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.302387 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.302388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.303348 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.303350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.304143 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.304145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.304776 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.304777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.305050 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.305054 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.305148 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.305208 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 02:17:34.305387 32565 raft_consensus.cc:482] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:34.305449 32463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cae8a8f9564f92baf7242122b0cf03" candidate_uuid: "9fb40e9366054c12949cd4278bbc7f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1" is_pre_election: true
I20260808 02:17:34.305449 32565 raft_consensus.cc:504] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.305790 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cae8a8f9564f92baf7242122b0cf03" candidate_uuid: "9fb40e9366054c12949cd4278bbc7f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a71157043bf4d4ea60661df405aaf02" is_pre_election: true
I20260808 02:17:34.305891 32075 raft_consensus.cc:2454] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb40e9366054c12949cd4278bbc7f3f in term 0.
I20260808 02:17:34.305892 32463 raft_consensus.cc:2454] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb40e9366054c12949cd4278bbc7f3f in term 0.
I20260808 02:17:34.306128 32565 leader_election.cc:288] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391), 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429)
I20260808 02:17:34.306126 32169 leader_election.cc:302] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a71157043bf4d4ea60661df405aaf02, 9fb40e9366054c12949cd4278bbc7f3f; no voters:
I20260808 02:17:34.306290 32563 raft_consensus.cc:2781] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:34.306334 32563 raft_consensus.cc:482] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:34.306388 32563 raft_consensus.cc:3037] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.307428 32563 raft_consensus.cc:504] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.307574 32563 leader_election.cc:288] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [CANDIDATE]: Term 1 election: Requested vote from peers 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391), 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:34.307823 32463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cae8a8f9564f92baf7242122b0cf03" candidate_uuid: "9fb40e9366054c12949cd4278bbc7f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1"
I20260808 02:17:34.307914 32463 raft_consensus.cc:3037] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.307895 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cae8a8f9564f92baf7242122b0cf03" candidate_uuid: "9fb40e9366054c12949cd4278bbc7f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a71157043bf4d4ea60661df405aaf02"
I20260808 02:17:34.307962 32075 raft_consensus.cc:3037] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.308768 32075 raft_consensus.cc:2454] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb40e9366054c12949cd4278bbc7f3f in term 1.
I20260808 02:17:34.308768 32463 raft_consensus.cc:2454] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb40e9366054c12949cd4278bbc7f3f in term 1.
I20260808 02:17:34.308943 32169 leader_election.cc:302] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a71157043bf4d4ea60661df405aaf02, 9fb40e9366054c12949cd4278bbc7f3f; no voters:
I20260808 02:17:34.309033 32563 raft_consensus.cc:2781] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:34.309262 32563 raft_consensus.cc:686] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 LEADER]: Becoming Leader. State: Replica: 9fb40e9366054c12949cd4278bbc7f3f, State: Running, Role: LEADER
I20260808 02:17:34.309376 32563 consensus_queue.cc:237] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.310179 32589 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: TxnStatusTablet state changed. Reason: New leader 9fb40e9366054c12949cd4278bbc7f3f. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.310271 32589 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:17:34.310400 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.306268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45391 (local address 127.28.177.131:33775)
0808 02:17:34.306372 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.306382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.306418 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.306690 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.306691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.306693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.306765 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.306767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.307718 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.307719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.308727 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.308730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.310129 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.310130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.310229 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.310233 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.310282 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.310321 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 02:17:34.310582 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.306488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:33775 (local address 127.28.177.129:45391)
0808 02:17:34.306548 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.306558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.306566 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:34.306587 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.306587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.306596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.306650 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.306853 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.307595 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.309414 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.310028 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.310109 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.310120 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.310450 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.310520 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:17:34.310755 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cae8a8f9564f92baf7242122b0cf03" candidate_uuid: "17b1cea48a24457796fc2d77733195e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a71157043bf4d4ea60661df405aaf02" is_pre_election: true
I20260808 02:17:34.310860 32075 raft_consensus.cc:2379] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17b1cea48a24457796fc2d77733195e1 in current term 1: Already voted for candidate 9fb40e9366054c12949cd4278bbc7f3f in this term.
I20260808 02:17:34.310878 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.306335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44429 (local address 127.28.177.131:54635)
0808 02:17:34.306602 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.306612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.306677 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.306828 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.306829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.306831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:34.306890 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.306891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.307610 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.307611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.309870 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.309873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.310624 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.310625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.310691 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.310694 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.310768 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.310805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":136,"threads_started":1}
I20260808 02:17:34.310871 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.306573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:54635 (local address 127.28.177.130:44429)
0808 02:17:34.306613 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.306615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.306664 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:17:34.306728 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.306729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.306737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.306790 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:34.306968 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.307486 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.309973 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.310540 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.310621 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.310628 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.310780 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.310820 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:17:34.311173 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cae8a8f9564f92baf7242122b0cf03" candidate_uuid: "17b1cea48a24457796fc2d77733195e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f" is_pre_election: true
I20260808 02:17:34.311183 32591 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:17:34.311344 32591 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:17:34.311300 31854 catalog_manager.cc:5495] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130). New cstate: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:34.311499 32367 leader_election.cc:302] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17b1cea48a24457796fc2d77733195e1; no voters: 3a71157043bf4d4ea60661df405aaf02, 9fb40e9366054c12949cd4278bbc7f3f
I20260808 02:17:34.311671 32565 raft_consensus.cc:2726] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:17:34.335366 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.332389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.28.177.130:44429)
0808 02:17:34.332476 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.332477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.332486 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:34.332543 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.332544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.332577 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.332641 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:34.332808 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.333367 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.334519 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.334730 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.334799 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.335026 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.335122 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.335188 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.335270 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.335288 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:17:34.335398 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.332320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44429 (local address 127.0.0.1:42048)
0808 02:17:34.332410 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.332422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.332467 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.332677 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.332678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.332680 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:34.332736 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.332739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.333456 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.333457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.334407 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.334410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.334818 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.334820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.334889 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.334907 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.335195 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.335203 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.335268 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.335315 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:17:34.337395 32463 raft_consensus.cc:1261] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Refusing update from remote peer 9fb40e9366054c12949cd4278bbc7f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:34.337404 32075 raft_consensus.cc:1261] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Refusing update from remote peer 9fb40e9366054c12949cd4278bbc7f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:34.337704 32563 consensus_queue.cc:1035] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:34.337898 32589 consensus_queue.cc:1035] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:34.339223 32568 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: TxnStatusTablet state changed. Reason: New leader 9fb40e9366054c12949cd4278bbc7f3f. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.339323 32568 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:34.339620 32565 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: TxnStatusTablet state changed. Reason: New leader 9fb40e9366054c12949cd4278bbc7f3f. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.339705 32565 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:34.340080 32589 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.340170 32589 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:17:34.340217 32589 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.340270 32589 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 9fb40e9366054c12949cd4278bbc7f3f: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:17:34.341151 32568 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.341241 32568 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 3a71157043bf4d4ea60661df405aaf02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:34.341421 32565 tablet_replica.cc:439] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fb40e9366054c12949cd4278bbc7f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } } }
I20260808 02:17:34.341498 32565 tablet_replica.cc:442] T a1cae8a8f9564f92baf7242122b0cf03 P 17b1cea48a24457796fc2d77733195e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:17:34.342417 32600 mvcc.cc:205] Tried to move back new op lower bound from 7316092740962717696 to 7316092740852281344. Current Snapshot: MvccSnapshot[applied={T|T < 7316092740962717696}]
I20260808 02:17:34.346830 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.343685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.28.177.129:45391)
0808 02:17:34.343727 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.343729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.343740 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:34.343825 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.343826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.343847 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.343927 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:34.344258 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.345067 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.345916 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.346232 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.346330 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.346476 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.346601 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.346685 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.346709 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.346733 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:34.353029 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.349777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44429 (local address 127.28.177.129:37505)
0808 02:17:34.349983 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.349997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.350071 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.350216 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.350217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.350219 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:34.350283 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.350285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.350920 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.350921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.351873 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.351876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.352478 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.352479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.352553 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.352571 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.352798 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.352800 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.352883 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.352932 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260808 02:17:34.353086 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.349826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:37505 (local address 127.28.177.130:44429)
0808 02:17:34.349878 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.349881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.350078 (+ 197us) server_negotiation.cc:407] Connection header received
0808 02:17:34.350128 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.350129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.350139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.350198 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:34.350349 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.350850 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.352070 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.352403 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.352493 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.352690 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.352724 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.352777 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.352898 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.352925 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:17:34.355981 32463 raft_consensus.cc:1261] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Refusing update from remote peer 3a71157043bf4d4ea60661df405aaf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:34.356027 32270 raft_consensus.cc:1261] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Refusing update from remote peer 3a71157043bf4d4ea60661df405aaf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:34.356261 32570 consensus_queue.cc:1035] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:34.356410 32568 consensus_queue.cc:1035] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:34.358193 32597 mvcc.cc:205] Tried to move back new op lower bound from 7316092741040181248 to 7316092740131627008. Current Snapshot: MvccSnapshot[applied={T|T < 7316092741040181248}]
I20260808 02:17:34.366303 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.362873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45391 (local address 127.28.177.130:41765)
0808 02:17:34.363060 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:34.363072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:34.363143 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:34.363274 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:34.363275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:34.363276 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:34.363348 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.363349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.364283 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.364284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.365276 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:34.365284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.365774 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.365774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:34.365831 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.365846 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.366133 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.366135 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:17:34.366184 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:17:34.366223 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260808 02:17:34.366298 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:34.362916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:41765 (local address 127.28.177.129:45391)
0808 02:17:34.362958 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:34.362960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:34.363147 (+ 187us) server_negotiation.cc:407] Connection header received
0808 02:17:34.363203 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:34.363203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:34.363214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:34.363272 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:34.363407 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.364148 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.365364 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:34.365690 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:34.365768 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:34.365927 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:34.365961 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:34.366015 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:34.366204 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:17:34.366226 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:17:34.375689 31853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:34.375825 31853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:34.376711 31853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Sending DeleteTablet for 3 replicas of tablet 186df0d418d342298007769d72acc0df
I20260808 02:17:34.377149 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet 186df0d418d342298007769d72acc0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:34 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:56450
I20260808 02:17:34.377156 32438 tablet_service.cc:1514] Processing DeleteTablet for tablet 186df0d418d342298007769d72acc0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:34 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:50920
I20260808 02:17:34.377592 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet 186df0d418d342298007769d72acc0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:34 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42038
I20260808 02:17:34.377656 31853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:34.377822 31853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:34.377840 32628 tablet_replica.cc:330] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: stopping tablet replica
I20260808 02:17:34.377944 32628 raft_consensus.cc:2229] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:34.378362 32628 raft_consensus.cc:2258] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:34.379351 32044 tablet_service.cc:1467] Processing CreateTablet for tablet a3001ba023bd4ffcad4928ee1a5ec3cb (DEFAULT_TABLE table=test-table [id=7b5f828769894d2f9995293ae18ee10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:34.379344 32243 tablet_service.cc:1467] Processing CreateTablet for tablet a3001ba023bd4ffcad4928ee1a5ec3cb (DEFAULT_TABLE table=test-table [id=7b5f828769894d2f9995293ae18ee10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:34.379488 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3001ba023bd4ffcad4928ee1a5ec3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.379498 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3001ba023bd4ffcad4928ee1a5ec3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.381049 32558 tablet_bootstrap.cc:492] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Bootstrap starting.
I20260808 02:17:34.381050 32564 tablet_bootstrap.cc:492] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Bootstrap starting.
I20260808 02:17:34.381534 32558 tablet_bootstrap.cc:654] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.381534 32564 tablet_bootstrap.cc:654] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.382350 32438 tablet_service.cc:1467] Processing CreateTablet for tablet a3001ba023bd4ffcad4928ee1a5ec3cb (DEFAULT_TABLE table=test-table [id=7b5f828769894d2f9995293ae18ee10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:34.382580 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3001ba023bd4ffcad4928ee1a5ec3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:34.382755 32628 ts_tablet_manager.cc:1905] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:34.384208 32560 tablet_bootstrap.cc:492] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Bootstrap starting.
I20260808 02:17:34.384690 32560 tablet_bootstrap.cc:654] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:34.384660 32628 ts_tablet_manager.cc:1918] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:34.384754 32628 log.cc:1196] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/186df0d418d342298007769d72acc0df
I20260808 02:17:34.385035 32626 tablet_replica.cc:330] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: stopping tablet replica
I20260808 02:17:34.385187 32626 raft_consensus.cc:2229] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:17:34.385344 32626 raft_consensus.cc:2258] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:34.385478 32564 tablet_bootstrap.cc:492] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: No bootstrap required, opened a new log
I20260808 02:17:34.385537 32564 ts_tablet_manager.cc:1397] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260808 02:17:34.385706 32564 raft_consensus.cc:348] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.385807 32564 raft_consensus.cc:374] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.385840 32564 raft_consensus.cc:729] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.385886 32564 consensus_queue.cc:260] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.385901 32626 ts_tablet_manager.cc:1905] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:34.386005 32564 ts_tablet_manager.cc:1428] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:17:34.386747 32628 ts_tablet_manager.cc:1939] T 186df0d418d342298007769d72acc0df P 9fb40e9366054c12949cd4278bbc7f3f: Deleting consensus metadata
I20260808 02:17:34.387244 31824 catalog_manager.cc:4841] TS 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429): tablet 186df0d418d342298007769d72acc0df (table test-table [id=81f484c7ddb840699e82e964b639bce8]) successfully deleted
I20260808 02:17:34.387540 32560 tablet_bootstrap.cc:492] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: No bootstrap required, opened a new log
I20260808 02:17:34.387619 32560 ts_tablet_manager.cc:1397] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 02:17:34.387794 32560 raft_consensus.cc:348] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.387813 32626 ts_tablet_manager.cc:1918] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:34.387880 32560 raft_consensus.cc:374] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.387912 32626 log.cc:1196] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/186df0d418d342298007769d72acc0df
I20260808 02:17:34.387928 32560 raft_consensus.cc:729] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b1cea48a24457796fc2d77733195e1, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.387993 32560 consensus_queue.cc:260] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.388103 32560 ts_tablet_manager.cc:1428] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:17:34.388216 32626 ts_tablet_manager.cc:1939] T 186df0d418d342298007769d72acc0df P 3a71157043bf4d4ea60661df405aaf02: Deleting consensus metadata
I20260808 02:17:34.388665 32627 tablet_replica.cc:330] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: stopping tablet replica
I20260808 02:17:34.388661 31825 catalog_manager.cc:4841] TS 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391): tablet 186df0d418d342298007769d72acc0df (table test-table [id=81f484c7ddb840699e82e964b639bce8]) successfully deleted
I20260808 02:17:34.388792 32627 raft_consensus.cc:2229] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:34.388911 32627 raft_consensus.cc:2258] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:34.389389 32558 tablet_bootstrap.cc:492] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: No bootstrap required, opened a new log
I20260808 02:17:34.389472 32558 ts_tablet_manager.cc:1397] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260808 02:17:34.389649 32558 raft_consensus.cc:348] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.389734 32558 raft_consensus.cc:374] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:34.389766 32558 raft_consensus.cc:729] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb40e9366054c12949cd4278bbc7f3f, State: Initialized, Role: FOLLOWER
I20260808 02:17:34.389814 32558 consensus_queue.cc:260] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.389926 32558 ts_tablet_manager.cc:1428] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:34.393232 32627 ts_tablet_manager.cc:1905] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:34.395179 32627 ts_tablet_manager.cc:1918] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:34.395280 32627 log.cc:1196] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/186df0d418d342298007769d72acc0df
I20260808 02:17:34.395843 32627 ts_tablet_manager.cc:1939] T 186df0d418d342298007769d72acc0df P 17b1cea48a24457796fc2d77733195e1: Deleting consensus metadata
I20260808 02:17:34.396320 31827 catalog_manager.cc:4841] TS 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015): tablet 186df0d418d342298007769d72acc0df (table test-table [id=81f484c7ddb840699e82e964b639bce8]) successfully deleted
I20260808 02:17:34.478552 32570 raft_consensus.cc:482] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:34.478664 32570 raft_consensus.cc:504] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.478849 32570 leader_election.cc:288] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429), 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:34.479127 32463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3001ba023bd4ffcad4928ee1a5ec3cb" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1" is_pre_election: true
I20260808 02:17:34.479144 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3001ba023bd4ffcad4928ee1a5ec3cb" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f" is_pre_election: true
I20260808 02:17:34.479242 32270 raft_consensus.cc:2454] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 0.
I20260808 02:17:34.479245 32463 raft_consensus.cc:2454] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 0.
I20260808 02:17:34.479499 31972 leader_election.cc:302] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a71157043bf4d4ea60661df405aaf02, 9fb40e9366054c12949cd4278bbc7f3f; no voters:
I20260808 02:17:34.479681 32570 raft_consensus.cc:2781] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:34.479727 32570 raft_consensus.cc:482] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:34.479748 32570 raft_consensus.cc:3037] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.480688 32570 raft_consensus.cc:504] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.481009 32463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3001ba023bd4ffcad4928ee1a5ec3cb" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1"
I20260808 02:17:34.481024 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3001ba023bd4ffcad4928ee1a5ec3cb" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f"
I20260808 02:17:34.481096 32463 raft_consensus.cc:3037] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.481094 32270 raft_consensus.cc:3037] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:34.481860 32270 raft_consensus.cc:2454] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 1.
I20260808 02:17:34.481957 32463 raft_consensus.cc:2454] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 1.
I20260808 02:17:34.480821 32570 leader_election.cc:288] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 election: Requested vote from peers 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429), 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:34.482116 31972 leader_election.cc:302] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a71157043bf4d4ea60661df405aaf02, 9fb40e9366054c12949cd4278bbc7f3f; no voters:
I20260808 02:17:34.482378 32570 raft_consensus.cc:2781] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:34.482430 32570 raft_consensus.cc:686] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 1 LEADER]: Becoming Leader. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Running, Role: LEADER
I20260808 02:17:34.482479 32570 consensus_queue.cc:237] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:34.483053 31853 catalog_manager.cc:5495] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "3a71157043bf4d4ea60661df405aaf02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:34.509117 32463 raft_consensus.cc:1261] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Refusing update from remote peer 3a71157043bf4d4ea60661df405aaf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:34.509191 32270 raft_consensus.cc:1261] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Refusing update from remote peer 3a71157043bf4d4ea60661df405aaf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:34.509411 32624 consensus_queue.cc:1035] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:34.509544 32570 consensus_queue.cc:1035] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:34.511188 32595 mvcc.cc:205] Tried to move back new op lower bound from 7316092741666856960 to 7316092741560504320. Current Snapshot: MvccSnapshot[applied={T|T < 7316092741666856960}]
W20260808 02:17:34.515627 32597 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:17:36.038915 31853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:36.039023 31853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:36.040258 31853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Sending DeleteTablet for 3 replicas of tablet a3001ba023bd4ffcad4928ee1a5ec3cb
I20260808 02:17:36.040588 32438 tablet_service.cc:1514] Processing DeleteTablet for tablet a3001ba023bd4ffcad4928ee1a5ec3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:36 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:50920
I20260808 02:17:36.040591 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet a3001ba023bd4ffcad4928ee1a5ec3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:36 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:56450
I20260808 02:17:36.040611 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet a3001ba023bd4ffcad4928ee1a5ec3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:36 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42038
I20260808 02:17:36.040887 32666 tablet_replica.cc:330] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: stopping tablet replica
I20260808 02:17:36.040913 32663 tablet_replica.cc:330] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: stopping tablet replica
I20260808 02:17:36.040968 32663 raft_consensus.cc:2229] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:17:36.041044 32663 raft_consensus.cc:2258] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:36.041056 32666 raft_consensus.cc:2229] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:36.041158 32662 tablet_replica.cc:330] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: stopping tablet replica
I20260808 02:17:36.041225 32662 raft_consensus.cc:2229] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:36.041266 32662 raft_consensus.cc:2258] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:36.041353 32663 ts_tablet_manager.cc:1905] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:36.041575 32662 ts_tablet_manager.cc:1905] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:36.043124 32666 raft_consensus.cc:2258] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:36.043486 32666 ts_tablet_manager.cc:1905] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:36.043603 32662 ts_tablet_manager.cc:1918] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:17:36.043717 32662 log.cc:1196] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/a3001ba023bd4ffcad4928ee1a5ec3cb
I20260808 02:17:36.044047 32662 ts_tablet_manager.cc:1939] T a3001ba023bd4ffcad4928ee1a5ec3cb P 17b1cea48a24457796fc2d77733195e1: Deleting consensus metadata
I20260808 02:17:36.044289 32663 ts_tablet_manager.cc:1918] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:17:36.044353 32663 log.cc:1196] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/a3001ba023bd4ffcad4928ee1a5ec3cb
I20260808 02:17:36.044488 31827 catalog_manager.cc:4841] TS 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015): tablet a3001ba023bd4ffcad4928ee1a5ec3cb (table test-table [id=7b5f828769894d2f9995293ae18ee10e]) successfully deleted
I20260808 02:17:36.044641 32663 ts_tablet_manager.cc:1939] T a3001ba023bd4ffcad4928ee1a5ec3cb P 3a71157043bf4d4ea60661df405aaf02: Deleting consensus metadata
I20260808 02:17:36.045018 31825 catalog_manager.cc:4841] TS 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391): tablet a3001ba023bd4ffcad4928ee1a5ec3cb (table test-table [id=7b5f828769894d2f9995293ae18ee10e]) successfully deleted
I20260808 02:17:36.045372 32666 ts_tablet_manager.cc:1918] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:17:36.045455 32666 log.cc:1196] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/a3001ba023bd4ffcad4928ee1a5ec3cb
I20260808 02:17:36.045746 32666 ts_tablet_manager.cc:1939] T a3001ba023bd4ffcad4928ee1a5ec3cb P 9fb40e9366054c12949cd4278bbc7f3f: Deleting consensus metadata
I20260808 02:17:36.046228 31824 catalog_manager.cc:4841] TS 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429): tablet a3001ba023bd4ffcad4928ee1a5ec3cb (table test-table [id=7b5f828769894d2f9995293ae18ee10e]) successfully deleted
Aug 08 02:17:36 dist-test-slave-3h5h krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:17:36 dist-test-slave-3h5h krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:36.075160 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.060687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.28.177.190:45947)
0808 02:17:36.060878 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:36.060881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:36.060894 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:17:36.060976 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:36.060977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:36.060989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:36.061137 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:36.061798 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.062321 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.063103 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.063335 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.063389 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.071430 (+ 8041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:36.071432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:36.071433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:36.071443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:36.073410 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:36.073851 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:36.073852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:36.073852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:36.073917 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:36.074331 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:36.074332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:36.074332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:36.074470 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:36.074567 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:36.074870 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 02:17:36.074970 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260808 02:17:36.077205 31853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:36.077446 31853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:36.079052 32044 tablet_service.cc:1467] Processing CreateTablet for tablet c3b0f230df7b4dc98b620ed30b433f96 (DEFAULT_TABLE table=test-table [id=f165d78e58114d01a97fff11abf4440f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:36.079113 32438 tablet_service.cc:1467] Processing CreateTablet for tablet c3b0f230df7b4dc98b620ed30b433f96 (DEFAULT_TABLE table=test-table [id=f165d78e58114d01a97fff11abf4440f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:36.079085 32243 tablet_service.cc:1467] Processing CreateTablet for tablet c3b0f230df7b4dc98b620ed30b433f96 (DEFAULT_TABLE table=test-table [id=f165d78e58114d01a97fff11abf4440f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:36.079219 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b0f230df7b4dc98b620ed30b433f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:36.079219 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b0f230df7b4dc98b620ed30b433f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:36.079275 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b0f230df7b4dc98b620ed30b433f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:36.080458 32678 tablet_bootstrap.cc:492] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Bootstrap starting.
I20260808 02:17:36.080657 32679 tablet_bootstrap.cc:492] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Bootstrap starting.
I20260808 02:17:36.080796 32680 tablet_bootstrap.cc:492] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Bootstrap starting.
I20260808 02:17:36.081051 32679 tablet_bootstrap.cc:654] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:36.081182 32678 tablet_bootstrap.cc:654] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:36.081218 32680 tablet_bootstrap.cc:654] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:36.081673 32679 tablet_bootstrap.cc:492] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: No bootstrap required, opened a new log
I20260808 02:17:36.081745 32679 ts_tablet_manager.cc:1397] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:36.081859 32680 tablet_bootstrap.cc:492] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: No bootstrap required, opened a new log
I20260808 02:17:36.081920 32680 ts_tablet_manager.cc:1397] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:36.081878 32679 raft_consensus.cc:348] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.082034 32679 raft_consensus.cc:374] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:36.082064 32679 raft_consensus.cc:729] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Initialized, Role: FOLLOWER
I20260808 02:17:36.082121 32679 consensus_queue.cc:260] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.082136 32680 raft_consensus.cc:348] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.082224 32680 raft_consensus.cc:374] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:36.082233 32679 ts_tablet_manager.cc:1428] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:36.082259 32680 raft_consensus.cc:729] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b1cea48a24457796fc2d77733195e1, State: Initialized, Role: FOLLOWER
I20260808 02:17:36.082549 32680 consensus_queue.cc:260] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.082727 32678 tablet_bootstrap.cc:492] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: No bootstrap required, opened a new log
I20260808 02:17:36.082782 32680 ts_tablet_manager.cc:1428] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:36.082791 32678 ts_tablet_manager.cc:1397] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:17:36.082926 32678 raft_consensus.cc:348] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.083009 32678 raft_consensus.cc:374] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:36.083024 32678 raft_consensus.cc:729] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb40e9366054c12949cd4278bbc7f3f, State: Initialized, Role: FOLLOWER
I20260808 02:17:36.083087 32678 consensus_queue.cc:260] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.083200 32678 ts_tablet_manager.cc:1428] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:36.094000 32685 raft_consensus.cc:482] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:36.094086 32685 raft_consensus.cc:504] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.094360 32685 leader_election.cc:288] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391), 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429)
I20260808 02:17:36.094481 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b0f230df7b4dc98b620ed30b433f96" candidate_uuid: "17b1cea48a24457796fc2d77733195e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a71157043bf4d4ea60661df405aaf02" is_pre_election: true
I20260808 02:17:36.094481 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b0f230df7b4dc98b620ed30b433f96" candidate_uuid: "17b1cea48a24457796fc2d77733195e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f" is_pre_election: true
I20260808 02:17:36.094599 32075 raft_consensus.cc:2454] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b1cea48a24457796fc2d77733195e1 in term 0.
I20260808 02:17:36.094600 32270 raft_consensus.cc:2454] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b1cea48a24457796fc2d77733195e1 in term 0.
I20260808 02:17:36.094784 32368 leader_election.cc:302] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b1cea48a24457796fc2d77733195e1, 3a71157043bf4d4ea60661df405aaf02; no voters:
I20260808 02:17:36.094973 32685 raft_consensus.cc:2781] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:36.095034 32685 raft_consensus.cc:482] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:36.095067 32685 raft_consensus.cc:3037] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:36.095803 32685 raft_consensus.cc:504] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.095952 32685 leader_election.cc:288] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [CANDIDATE]: Term 1 election: Requested vote from peers 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391), 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429)
I20260808 02:17:36.096119 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b0f230df7b4dc98b620ed30b433f96" candidate_uuid: "17b1cea48a24457796fc2d77733195e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a71157043bf4d4ea60661df405aaf02"
I20260808 02:17:36.096112 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b0f230df7b4dc98b620ed30b433f96" candidate_uuid: "17b1cea48a24457796fc2d77733195e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f"
I20260808 02:17:36.096191 32075 raft_consensus.cc:3037] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:36.096212 32270 raft_consensus.cc:3037] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:36.096905 32270 raft_consensus.cc:2454] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b1cea48a24457796fc2d77733195e1 in term 1.
I20260808 02:17:36.096910 32075 raft_consensus.cc:2454] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b1cea48a24457796fc2d77733195e1 in term 1.
I20260808 02:17:36.097064 32367 leader_election.cc:302] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b1cea48a24457796fc2d77733195e1, 9fb40e9366054c12949cd4278bbc7f3f; no voters:
I20260808 02:17:36.097150 32685 raft_consensus.cc:2781] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:36.097311 32685 raft_consensus.cc:686] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 1 LEADER]: Becoming Leader. State: Replica: 17b1cea48a24457796fc2d77733195e1, State: Running, Role: LEADER
I20260808 02:17:36.097404 32685 consensus_queue.cc:237] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.098201 31854 catalog_manager.cc:5495] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b1cea48a24457796fc2d77733195e1 (127.28.177.131). New cstate: current_term: 1 leader_uuid: "17b1cea48a24457796fc2d77733195e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } health_report { overall_health: HEALTHY } } }
I20260808 02:17:36.109354 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.106665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.28.177.131:44015)
0808 02:17:36.106815 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:36.106816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:36.106823 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:17:36.106866 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:36.106866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:36.106884 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:36.106979 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:36.107144 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.107724 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.108496 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.108716 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.108794 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.108966 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:36.109083 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:36.109189 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:36.109227 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:17:36.109252 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260808 02:17:36.114420 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.111723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:34949 (local address 127.28.177.130:44429)
0808 02:17:36.111875 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:36.111877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:36.111903 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:17:36.111971 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:36.111972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:36.111982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:36.112046 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:36.112157 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.112627 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.113555 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.113784 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.113847 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.114147 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:36.114176 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:36.114215 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:36.114340 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 02:17:36.114360 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":103,"threads_started":1}
I20260808 02:17:36.114445 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.111683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44429 (local address 127.28.177.131:34949)
0808 02:17:36.111849 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:36.111860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:36.111916 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:36.112048 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:36.112049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:36.112050 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:36.112099 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:36.112100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.112704 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.112705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:36.113460 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:36.113464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.113907 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.113909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:36.113970 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.114060 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:36.114273 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:36.114276 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:36.114331 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:17:36.114371 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":105,"threads_started":1}
I20260808 02:17:36.117089 32075 raft_consensus.cc:1261] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Refusing update from remote peer 17b1cea48a24457796fc2d77733195e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:36.117193 32270 raft_consensus.cc:1261] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Refusing update from remote peer 17b1cea48a24457796fc2d77733195e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:36.117415 32685 consensus_queue.cc:1035] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:36.117553 32686 consensus_queue.cc:1035] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:36.126077 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.123295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:46785 (local address 127.28.177.131:44015)
0808 02:17:36.123389 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:36.123391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:36.123475 (+ 84us) server_negotiation.cc:407] Connection header received
0808 02:17:36.123550 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:36.123551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:36.123565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:36.123626 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:36.123756 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.124280 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.125212 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.125536 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.125615 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.125788 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:36.125825 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:36.125906 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:36.125976 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:17:36.126000 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:17:36.126127 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.123232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44015 (local address 127.28.177.130:46785)
0808 02:17:36.123378 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:36.123390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:36.123490 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:36.123630 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:36.123631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:36.123633 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:17:36.123687 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:36.123688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.124383 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.124384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:36.125120 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:36.125122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.125614 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.125616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:36.125681 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.125689 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:17:36.125911 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:17:36.125913 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:17:36.125971 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:17:36.126032 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":92,"threads_started":1}
I20260808 02:17:36.136531 31854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:36.136657 31854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:36.137411 31854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Sending DeleteTablet for 3 replicas of tablet c3b0f230df7b4dc98b620ed30b433f96
I20260808 02:17:36.137737 32438 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b0f230df7b4dc98b620ed30b433f96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:36 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:50920
I20260808 02:17:36.137728 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b0f230df7b4dc98b620ed30b433f96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:36 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:56450
I20260808 02:17:36.137835 32662 tablet_replica.cc:330] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: stopping tablet replica
I20260808 02:17:36.137851 32663 tablet_replica.cc:330] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: stopping tablet replica
I20260808 02:17:36.137887 32662 raft_consensus.cc:2229] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:17:36.137892 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b0f230df7b4dc98b620ed30b433f96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:36 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42038
I20260808 02:17:36.137934 32663 raft_consensus.cc:2229] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:36.137977 32663 raft_consensus.cc:2258] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:36.137988 32662 raft_consensus.cc:2258] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:36.138006 32666 tablet_replica.cc:330] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: stopping tablet replica
I20260808 02:17:36.138065 32666 raft_consensus.cc:2229] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:36.138110 32666 raft_consensus.cc:2258] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:36.138356 32662 ts_tablet_manager.cc:1905] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:36.138388 32666 ts_tablet_manager.cc:1905] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:36.138612 32663 ts_tablet_manager.cc:1905] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:36.140292 31854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:17:36.140407 32663 ts_tablet_manager.cc:1918] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:36.140467 32663 log.cc:1196] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/c3b0f230df7b4dc98b620ed30b433f96
W20260808 02:17:36.140475 31854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:36.140702 32663 ts_tablet_manager.cc:1939] T c3b0f230df7b4dc98b620ed30b433f96 P 3a71157043bf4d4ea60661df405aaf02: Deleting consensus metadata
I20260808 02:17:36.140826 32666 ts_tablet_manager.cc:1918] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:36.140901 32666 log.cc:1196] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/c3b0f230df7b4dc98b620ed30b433f96
I20260808 02:17:36.141075 31825 catalog_manager.cc:4841] TS 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391): tablet c3b0f230df7b4dc98b620ed30b433f96 (table test-table [id=f165d78e58114d01a97fff11abf4440f]) successfully deleted
I20260808 02:17:36.141139 32666 ts_tablet_manager.cc:1939] T c3b0f230df7b4dc98b620ed30b433f96 P 9fb40e9366054c12949cd4278bbc7f3f: Deleting consensus metadata
I20260808 02:17:36.141227 32662 ts_tablet_manager.cc:1918] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:17:36.141284 32662 log.cc:1196] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/c3b0f230df7b4dc98b620ed30b433f96
I20260808 02:17:36.141471 31824 catalog_manager.cc:4841] TS 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429): tablet c3b0f230df7b4dc98b620ed30b433f96 (table test-table [id=f165d78e58114d01a97fff11abf4440f]) successfully deleted
I20260808 02:17:36.141979 32662 ts_tablet_manager.cc:1939] T c3b0f230df7b4dc98b620ed30b433f96 P 17b1cea48a24457796fc2d77733195e1: Deleting consensus metadata
I20260808 02:17:36.142061 32438 tablet_service.cc:1467] Processing CreateTablet for tablet fd2fe9b5bbca4a6fbfa5e6873654698e (DEFAULT_TABLE table=test-table [id=3f4d70ceee674e8e943ab762f8a8e2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:36.142174 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2fe9b5bbca4a6fbfa5e6873654698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:36.142305 32044 tablet_service.cc:1467] Processing CreateTablet for tablet fd2fe9b5bbca4a6fbfa5e6873654698e (DEFAULT_TABLE table=test-table [id=3f4d70ceee674e8e943ab762f8a8e2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:36.142310 31827 catalog_manager.cc:4841] TS 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015): tablet c3b0f230df7b4dc98b620ed30b433f96 (table test-table [id=f165d78e58114d01a97fff11abf4440f]) successfully deleted
I20260808 02:17:36.142393 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2fe9b5bbca4a6fbfa5e6873654698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:36.142611 32243 tablet_service.cc:1467] Processing CreateTablet for tablet fd2fe9b5bbca4a6fbfa5e6873654698e (DEFAULT_TABLE table=test-table [id=3f4d70ceee674e8e943ab762f8a8e2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:36.142733 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2fe9b5bbca4a6fbfa5e6873654698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:36.143776 32680 tablet_bootstrap.cc:492] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Bootstrap starting.
I20260808 02:17:36.143771 32679 tablet_bootstrap.cc:492] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02: Bootstrap starting.
I20260808 02:17:36.144145 32680 tablet_bootstrap.cc:654] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:36.144256 32679 tablet_bootstrap.cc:654] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:36.144307 32678 tablet_bootstrap.cc:492] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Bootstrap starting.
I20260808 02:17:36.144654 32678 tablet_bootstrap.cc:654] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:36.145134 32680 tablet_bootstrap.cc:492] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: No bootstrap required, opened a new log
I20260808 02:17:36.145193 32680 ts_tablet_manager.cc:1397] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:36.145268 32678 tablet_bootstrap.cc:492] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: No bootstrap required, opened a new log
I20260808 02:17:36.145358 32680 raft_consensus.cc:348] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.145380 32678 ts_tablet_manager.cc:1397] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:36.145434 32680 raft_consensus.cc:374] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:36.145457 32680 raft_consensus.cc:729] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b1cea48a24457796fc2d77733195e1, State: Initialized, Role: FOLLOWER
I20260808 02:17:36.145504 32680 consensus_queue.cc:260] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.145538 32678 raft_consensus.cc:348] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.145603 32680 ts_tablet_manager.cc:1428] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:36.145618 32678 raft_consensus.cc:374] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:36.145650 32678 raft_consensus.cc:729] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb40e9366054c12949cd4278bbc7f3f, State: Initialized, Role: FOLLOWER
I20260808 02:17:36.145699 32678 consensus_queue.cc:260] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.145793 32678 ts_tablet_manager.cc:1428] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:36.149317 32679 tablet_bootstrap.cc:492] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02: No bootstrap required, opened a new log
I20260808 02:17:36.149400 32679 ts_tablet_manager.cc:1397] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260808 02:17:36.149536 32679 raft_consensus.cc:348] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.149636 32679 raft_consensus.cc:374] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:36.149673 32679 raft_consensus.cc:729] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Initialized, Role: FOLLOWER
I20260808 02:17:36.149730 32679 consensus_queue.cc:260] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.149847 32679 ts_tablet_manager.cc:1428] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:36.410959 32638 raft_consensus.cc:482] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:36.411188 32638 raft_consensus.cc:504] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.411441 32638 leader_election.cc:288] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429), 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:36.411842 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2fe9b5bbca4a6fbfa5e6873654698e" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f" is_pre_election: true
I20260808 02:17:36.411978 32270 raft_consensus.cc:2454] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 0.
I20260808 02:17:36.411844 32465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2fe9b5bbca4a6fbfa5e6873654698e" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1" is_pre_election: true
I20260808 02:17:36.412084 32465 raft_consensus.cc:2454] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 0.
I20260808 02:17:36.412328 31974 leader_election.cc:302] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b1cea48a24457796fc2d77733195e1, 3a71157043bf4d4ea60661df405aaf02; no voters:
I20260808 02:17:36.412497 32638 raft_consensus.cc:2781] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:36.412541 32638 raft_consensus.cc:482] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:36.412567 32638 raft_consensus.cc:3037] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:36.413497 32638 raft_consensus.cc:504] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.413667 32638 leader_election.cc:288] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 election: Requested vote from peers 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429), 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015)
I20260808 02:17:36.413901 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2fe9b5bbca4a6fbfa5e6873654698e" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb40e9366054c12949cd4278bbc7f3f"
I20260808 02:17:36.413935 32465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2fe9b5bbca4a6fbfa5e6873654698e" candidate_uuid: "3a71157043bf4d4ea60661df405aaf02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b1cea48a24457796fc2d77733195e1"
I20260808 02:17:36.414016 32465 raft_consensus.cc:3037] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:36.414026 32270 raft_consensus.cc:3037] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:36.414891 32270 raft_consensus.cc:2454] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 1.
I20260808 02:17:36.414891 32465 raft_consensus.cc:2454] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a71157043bf4d4ea60661df405aaf02 in term 1.
I20260808 02:17:36.415148 31972 leader_election.cc:302] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a71157043bf4d4ea60661df405aaf02, 9fb40e9366054c12949cd4278bbc7f3f; no voters:
I20260808 02:17:36.415287 32638 raft_consensus.cc:2781] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:36.415392 32638 raft_consensus.cc:686] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [term 1 LEADER]: Becoming Leader. State: Replica: 3a71157043bf4d4ea60661df405aaf02, State: Running, Role: LEADER
I20260808 02:17:36.415459 32638 consensus_queue.cc:237] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } }
I20260808 02:17:36.416267 31854 catalog_manager.cc:5495] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "3a71157043bf4d4ea60661df405aaf02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a71157043bf4d4ea60661df405aaf02" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:36.435374 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:36.432438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.28.177.129:45391)
0808 02:17:36.432727 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:36.432729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:36.432738 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:36.432760 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:36.432761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:36.432772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:36.432905 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:36.433153 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.433650 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.434497 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:36.434735 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:36.434794 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:36.434954 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:36.434994 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:36.435061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:36.435197 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 02:17:36.435220 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":193,"threads_started":1}
I20260808 02:17:36.437850 32465 raft_consensus.cc:1261] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Refusing update from remote peer 3a71157043bf4d4ea60661df405aaf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:36.438207 32270 raft_consensus.cc:1261] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Refusing update from remote peer 3a71157043bf4d4ea60661df405aaf02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:36.438215 32638 consensus_queue.cc:1035] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b1cea48a24457796fc2d77733195e1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:36.438424 32638 consensus_queue.cc:1035] T fd2fe9b5bbca4a6fbfa5e6873654698e P 3a71157043bf4d4ea60661df405aaf02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb40e9366054c12949cd4278bbc7f3f" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:17:36.442948 32645 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:17:37.962736 31854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:37.962862 31854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:37.963905 31854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a783be595e604e278579ca746a56f5c3: Sending DeleteTablet for 3 replicas of tablet fd2fe9b5bbca4a6fbfa5e6873654698e
I20260808 02:17:37.964236 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2fe9b5bbca4a6fbfa5e6873654698e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:37 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42038
I20260808 02:17:37.964357 32438 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2fe9b5bbca4a6fbfa5e6873654698e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:37 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:50920
I20260808 02:17:37.964483 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 31953
I20260808 02:17:37.964535 32747 tablet_replica.cc:330] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: stopping tablet replica
I20260808 02:17:37.964579 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2fe9b5bbca4a6fbfa5e6873654698e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:37 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:56450
I20260808 02:17:37.969666 32748 tablet_replica.cc:330] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: stopping tablet replica
I20260808 02:17:37.969764 32748 raft_consensus.cc:2229] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:37.969827 32748 raft_consensus.cc:2258] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:37.970157 32748 ts_tablet_manager.cc:1905] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:37.964607 32747 raft_consensus.cc:2229] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:37.971197 32747 raft_consensus.cc:2258] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:37.971539 32747 ts_tablet_manager.cc:1905] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:37.972458 32748 ts_tablet_manager.cc:1918] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:17:37.972539 32748 log.cc:1196] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/fd2fe9b5bbca4a6fbfa5e6873654698e
I20260808 02:17:37.972805 32748 ts_tablet_manager.cc:1939] T fd2fe9b5bbca4a6fbfa5e6873654698e P 17b1cea48a24457796fc2d77733195e1: Deleting consensus metadata
I20260808 02:17:37.973161 32747 ts_tablet_manager.cc:1918] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:17:37.973249 32747 log.cc:1196] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/fd2fe9b5bbca4a6fbfa5e6873654698e
I20260808 02:17:37.973511 32747 ts_tablet_manager.cc:1939] T fd2fe9b5bbca4a6fbfa5e6873654698e P 9fb40e9366054c12949cd4278bbc7f3f: Deleting consensus metadata
I20260808 02:17:37.974090 31824 catalog_manager.cc:4841] TS 9fb40e9366054c12949cd4278bbc7f3f (127.28.177.130:44429): tablet fd2fe9b5bbca4a6fbfa5e6873654698e (table test-table [id=3f4d70ceee674e8e943ab762f8a8e2b9]) successfully deleted
I20260808 02:17:37.977293 31827 catalog_manager.cc:4841] TS 17b1cea48a24457796fc2d77733195e1 (127.28.177.131:44015): tablet fd2fe9b5bbca4a6fbfa5e6873654698e (table test-table [id=3f4d70ceee674e8e943ab762f8a8e2b9]) successfully deleted
W20260808 02:17:37.977813 31825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:45391 (error 108)
I20260808 02:17:37.978287 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 32147
I20260808 02:17:37.983788 32750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:37.978319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45391 (local address 127.0.0.1:56490)
0808 02:17:37.983637 (+ 5318us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:37.983719 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5278,"thread_start_us":106,"threads_started":1}
W20260808 02:17:37.984026 31825 catalog_manager.cc:4584] TS 3a71157043bf4d4ea60661df405aaf02 (127.28.177.129:45391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd2fe9b5bbca4a6fbfa5e6873654698e: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45391: connect: Connection refused (error 111)
I20260808 02:17:37.990567 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 32347
I20260808 02:17:38.002904 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 31800
2026-08-08T02:17:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6134 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:38 dist-test-slave-3h5h krb5kdc[32754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:38 dist-test-slave-3h5h krb5kdc[32754](info): set up 2 sockets
Aug 08 02:17:38 dist-test-slave-3h5h krb5kdc[32754](info): commencing operation
krb5kdc: starting...
W20260808 02:17:39.332592 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:39.473387 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:34739
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:39.561894 306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:39.562150 306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:39.562201 306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:39.563411 306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:39.563481 306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:39.563508 306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:39.563539 306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:39.563565 306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:39.565227 306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:34739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:39.565518 306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:39.565761 306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:39.571904 313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:39.571966 306 server_base.cc:1034] running on GCE node
W20260808 02:17:39.571952 320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:39.572082 314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:39.572454 306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:39.572718 306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:39.573907 306 hybrid_clock.cc:648] HybridClock initialized: now 1786155459573923 us; error 103 us; skew 500 ppm
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:39.576570 306 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:39.577116 306 webserver.cc:460] Webserver started at http://127.28.177.190:43269/ using document root <none> and password file <none>
I20260808 02:17:39.577348 306 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:39.577392 306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:39.577486 306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:39.578534 306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "8cbbcf9e6aef4030b28b52dd910bc21f"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "de8d08a5aa21c240b74d203bdb125e3e"
server_key_iv: "4a75a23a9a0fad44af9cbc8ef8f80892"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.578845 306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "8cbbcf9e6aef4030b28b52dd910bc21f"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "de8d08a5aa21c240b74d203bdb125e3e"
server_key_iv: "4a75a23a9a0fad44af9cbc8ef8f80892"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.580264 306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:39.581121 329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.581348 306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:39.581447 306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "8cbbcf9e6aef4030b28b52dd910bc21f"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "de8d08a5aa21c240b74d203bdb125e3e"
server_key_iv: "4a75a23a9a0fad44af9cbc8ef8f80892"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.581539 306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:39.595425 306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:39.598007 306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:39.598143 306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:39.602768 306 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:34739
I20260808 02:17:39.602849 396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:34739 every 8 connection(s)
I20260808 02:17:39.603257 306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:39.604353 397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:39.607595 397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f: Bootstrap starting.
I20260808 02:17:39.608220 397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:39.608501 397 log.cc:824] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:39.609503 397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f: No bootstrap required, opened a new log
I20260808 02:17:39.610184 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 306
I20260808 02:17:39.610263 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:39.610461 29382 external_mini_cluster.cc:1428] Setting key f4a7228f800be86a9d670a11f1387414
I20260808 02:17:39.612042 397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34739 } }
I20260808 02:17:39.612248 397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:39.612303 397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbbcf9e6aef4030b28b52dd910bc21f, State: Initialized, Role: FOLLOWER
I20260808 02:17:39.612511 397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34739 } }
I20260808 02:17:39.612622 397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:39.612663 397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:39.612723 397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:39.613574 397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34739 } }
I20260808 02:17:39.613713 397 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbbcf9e6aef4030b28b52dd910bc21f; no voters:
I20260808 02:17:39.613960 397 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:39.613991 404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:39.614128 404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [term 1 LEADER]: Becoming Leader. State: Replica: 8cbbcf9e6aef4030b28b52dd910bc21f, State: Running, Role: LEADER
I20260808 02:17:39.614283 397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:39.614288 404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34739 } }
I20260808 02:17:39.615860 405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34739 } } }
I20260808 02:17:39.616073 406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbbcf9e6aef4030b28b52dd910bc21f. Latest consensus state: current_term: 1 leader_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbbcf9e6aef4030b28b52dd910bc21f" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34739 } } }
I20260808 02:17:39.616155 405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [sys.catalog]: This master's current role is: LEADER
W20260808 02:17:39.616469 422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 02:17:39.616911 406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f [sys.catalog]: This master's current role is: LEADER
W20260808 02:17:39.616941 422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:17:39.617100 427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:39.617372 427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:39.619024 427 catalog_manager.cc:1347] Generated new cluster ID: 271d3cbe9df8494b9ec64aed4410d56e
I20260808 02:17:39.619063 427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:39.624022 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:39.611040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.28.177.190:34739)
0808 02:17:39.611379 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:39.611393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:39.611419 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:17:39.612112 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:39.612120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:39.612186 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:39.612358 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:39.613349 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.614512 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.615626 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.616265 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.616378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:39.619581 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:39.619585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:39.619586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:39.619599 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:39.621745 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:39.622163 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:39.622167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:39.622167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:39.622280 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:39.622530 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:39.622531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:39.622532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:39.622862 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:39.623054 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:39.623648 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 02:17:39.623798 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":250,"threads_started":1}
I20260808 02:17:39.631388 427 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:39.631964 427 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:39.634235 427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbbcf9e6aef4030b28b52dd910bc21f: Generated new TSK 0
I20260808 02:17:39.634426 427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:39.672888 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34739
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:39.762053 431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:39.762261 431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:39.762286 431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:39.763574 431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:39.763623 431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:39.763650 431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:39.765242 431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:39.765504 431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:39.765758 431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:39.771162 437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:39.771301 438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:39.771312 431 server_base.cc:1034] running on GCE node
W20260808 02:17:39.771166 440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:39.771694 431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:39.771971 431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:39.773200 431 hybrid_clock.cc:648] HybridClock initialized: now 1786155459773171 us; error 38 us; skew 500 ppm
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:39.776046 431 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:39.776616 431 webserver.cc:460] Webserver started at http://127.28.177.129:40609/ using document root <none> and password file <none>
I20260808 02:17:39.776963 431 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:39.777045 431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:39.777228 431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:39.778419 431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "7b5877f6fc1d4b50b071b6fd28f20e6e"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "0e6bc0f18e67c6294b9654c4ed5f22eb"
server_key_iv: "6e9a5015b9f9206730d707581d95c496"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.778862 431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "7b5877f6fc1d4b50b071b6fd28f20e6e"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "0e6bc0f18e67c6294b9654c4ed5f22eb"
server_key_iv: "6e9a5015b9f9206730d707581d95c496"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.780371 431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:39.781452 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.781796 431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:39.781977 431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "7b5877f6fc1d4b50b071b6fd28f20e6e"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "0e6bc0f18e67c6294b9654c4ed5f22eb"
server_key_iv: "6e9a5015b9f9206730d707581d95c496"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.782079 431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:39.805686 431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:39.807986 431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:39.808270 431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:39.808554 431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:39.808985 431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:39.809056 431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.809104 431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:39.809134 431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.815986 431 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:36927
I20260808 02:17:39.816035 606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:36927 every 8 connection(s)
I20260808 02:17:39.816489 431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:39.819576 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 431
I20260808 02:17:39.819696 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:39.819880 29382 external_mini_cluster.cc:1428] Setting key 2441eadba44dec0361bc7eeec77508c1
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:39.828424 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:39.817378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53267 (local address 127.28.177.190:34739)
0808 02:17:39.817453 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:39.817455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:39.817693 (+ 238us) server_negotiation.cc:407] Connection header received
0808 02:17:39.818223 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:39.818224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:39.818235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:39.818299 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:39.819341 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.819976 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.821629 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.821830 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.821888 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:39.824950 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:39.824952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:39.824952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:39.824960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:39.826644 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:39.827277 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:39.827278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:39.827278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:39.827339 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:39.827674 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:39.827675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:39.827675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:39.827800 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:39.827863 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:39.828268 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 02:17:39.828349 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:17:39.828758 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:39.817365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34739 (local address 127.28.177.129:53267)
0808 02:17:39.817549 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:39.817597 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:39.818129 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:39.818360 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:39.818364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:39.819003 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:39.819235 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:39.819240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.820102 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.820104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:39.821532 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:39.821536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.821924 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.821925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:39.822090 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:39.822902 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:39.822912 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:39.824795 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:39.826748 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:39.826757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:39.826760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:39.827178 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:39.827415 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:39.827416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:39.827416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:39.827588 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:39.827880 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:39.827888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:39.828099 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 02:17:39.828241 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 02:17:39.828445 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260808 02:17:39.829490 608 heartbeater.cc:347] Connected to a master server at 127.28.177.190:34739
I20260808 02:17:39.829619 608 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:39.829846 608 heartbeater.cc:511] Master 127.28.177.190:34739 requested a full tablet report, sending...
I20260808 02:17:39.830554 353 ts_manager.cc:194] Registered new tserver with Master: 7b5877f6fc1d4b50b071b6fd28f20e6e (127.28.177.129:36927)
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:17:39.831384 353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:53267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:39.854152 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34739
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:39.944813 615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:39.945113 615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:39.945158 615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:39.946420 615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:39.946506 615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:39.946548 615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:39.948115 615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:39.948411 615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:39.948699 615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:39.954596 630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:39.954627 622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:39.954663 615 server_base.cc:1034] running on GCE node
W20260808 02:17:39.954627 623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:39.955114 615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:39.955428 615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:39.956610 615 hybrid_clock.cc:648] HybridClock initialized: now 1786155459956589 us; error 51 us; skew 500 ppm
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:39.959590 615 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:39.960019 615 webserver.cc:460] Webserver started at http://127.28.177.130:40125/ using document root <none> and password file <none>
I20260808 02:17:39.960321 615 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:39.960371 615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:39.960474 615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:39.961644 615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "b831e90aaa6e4ecaa7c0f349bd4c758c"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "799f346cf7397581b9083063b46d9da9"
server_key_iv: "3268bb5dec2f8c3139181d76cafe2444"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.962026 615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "b831e90aaa6e4ecaa7c0f349bd4c758c"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "799f346cf7397581b9083063b46d9da9"
server_key_iv: "3268bb5dec2f8c3139181d76cafe2444"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.963472 615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:17:39.964373 637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.964589 615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:39.964690 615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "b831e90aaa6e4ecaa7c0f349bd4c758c"
format_stamp: "Formatted at 2026-08-08 02:17:39 on dist-test-slave-3h5h"
server_key: "799f346cf7397581b9083063b46d9da9"
server_key_iv: "3268bb5dec2f8c3139181d76cafe2444"
server_key_version: "encryptionkey@0"
I20260808 02:17:39.964742 615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:39.974568 615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:39.977106 615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:39.977240 615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:39.977447 615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:39.977819 615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:39.977859 615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.977882 615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:39.977895 615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:39.983880 615 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:38621
I20260808 02:17:39.983960 798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:38621 every 8 connection(s)
I20260808 02:17:39.984268 615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:17:39.990841 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 615
Aug 08 02:17:39 dist-test-slave-3h5h krb5kdc[32754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:39.990931 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:39.991175 29382 external_mini_cluster.cc:1428] Setting key 53b51e46dd135fab93221a499e47b783
I20260808 02:17:39.996153 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:39.985091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:42385 (local address 127.28.177.190:34739)
0808 02:17:39.985149 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:39.985151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:39.985500 (+ 349us) server_negotiation.cc:407] Connection header received
0808 02:17:39.986067 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:39.986068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:39.986083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:39.986146 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:39.987193 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.987711 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.988755 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.989044 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.989138 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:39.991532 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:39.991533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:39.991534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:39.991544 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:39.994251 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:39.994880 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:39.994881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:39.994882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:39.994949 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:39.995292 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:39.995294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:39.995295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:39.995469 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:39.995560 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:39.995911 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 02:17:39.996013 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:39.996188 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:39.985095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34739 (local address 127.28.177.130:42385)
0808 02:17:39.985340 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:39.985388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:39.985951 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:39.986212 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:39.986216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:39.986874 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:39.987080 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:39.987085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.987825 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.987827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:39.988655 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:39.988658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:39.989146 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:39.989147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:39.989282 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:39.989823 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:39.989832 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:39.991425 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:39.994346 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:39.994354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:39.994357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:39.994754 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:39.995028 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:39.995029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:39.995030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:39.995192 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:39.995557 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:39.995563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:39.995728 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 02:17:39.995856 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:17:39.996000 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":126,"threads_started":1}
I20260808 02:17:39.996815 799 heartbeater.cc:347] Connected to a master server at 127.28.177.190:34739
I20260808 02:17:39.996945 799 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:39.997151 799 heartbeater.cc:511] Master 127.28.177.190:34739 requested a full tablet report, sending...
I20260808 02:17:39.997632 353 ts_manager.cc:194] Registered new tserver with Master: b831e90aaa6e4ecaa7c0f349bd4c758c (127.28.177.130:38621)
I20260808 02:17:39.998174 353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:42385
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:40.024813 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34739
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:40.112897 808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:40.113116 808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:40.113140 808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:40.114873 808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:40.114923 808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:40.114950 808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:40.116813 808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:40.117156 808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:40.117431 808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:40.122761 819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:40.122812 814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:40.122773 815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:40.123097 808 server_base.cc:1034] running on GCE node
I20260808 02:17:40.123370 808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:40.123595 808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:40.124778 808 hybrid_clock.cc:648] HybridClock initialized: now 1786155460124768 us; error 36 us; skew 500 ppm
Aug 08 02:17:40 dist-test-slave-3h5h krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155460, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:40.127429 808 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:40.127894 808 webserver.cc:460] Webserver started at http://127.28.177.131:33251/ using document root <none> and password file <none>
I20260808 02:17:40.128142 808 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:40.128213 808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:40.128361 808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:40.129510 808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "dcab6828010c4dee9e09eb9e740bf30f"
format_stamp: "Formatted at 2026-08-08 02:17:40 on dist-test-slave-3h5h"
server_key: "6099c92415bc2e6cc4dbafcb8ef37966"
server_key_iv: "09002bd81575d48e52248e51544f227a"
server_key_version: "encryptionkey@0"
I20260808 02:17:40.129884 808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "dcab6828010c4dee9e09eb9e740bf30f"
format_stamp: "Formatted at 2026-08-08 02:17:40 on dist-test-slave-3h5h"
server_key: "6099c92415bc2e6cc4dbafcb8ef37966"
server_key_iv: "09002bd81575d48e52248e51544f227a"
server_key_version: "encryptionkey@0"
I20260808 02:17:40.131358 808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:40.132172 827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:40.132385 808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:40.132501 808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "dcab6828010c4dee9e09eb9e740bf30f"
format_stamp: "Formatted at 2026-08-08 02:17:40 on dist-test-slave-3h5h"
server_key: "6099c92415bc2e6cc4dbafcb8ef37966"
server_key_iv: "09002bd81575d48e52248e51544f227a"
server_key_version: "encryptionkey@0"
I20260808 02:17:40.132578 808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:40.144040 808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:40.146567 808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:40.146740 808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:40.146987 808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:40.147327 808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:40.147394 808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:40.147444 808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:40.147473 808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:40.153949 808 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:44273
I20260808 02:17:40.153996 985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:44273 every 8 connection(s)
I20260808 02:17:40.154383 808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:17:40 dist-test-slave-3h5h krb5kdc[32754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155460, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:40.161314 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 808
I20260808 02:17:40.161425 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:40.161655 29382 external_mini_cluster.cc:1428] Setting key 4ab3e30e3f960446eef185e1a4d9534c
I20260808 02:17:40.165027 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:40.155260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:41021 (local address 127.28.177.190:34739)
0808 02:17:40.155349 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:40.155351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:40.155522 (+ 171us) server_negotiation.cc:407] Connection header received
0808 02:17:40.156076 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:40.156078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:40.156091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:40.156173 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:40.157236 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:40.157762 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:40.158837 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:40.159118 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:40.159193 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:40.161816 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:40.161817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:40.161818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:40.161830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:40.163526 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:40.163944 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:40.163946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:40.163947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:40.164024 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:40.164304 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:40.164305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:40.164305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:40.164431 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:40.164510 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:40.164799 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 02:17:40.164915 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:17:40.165215 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:40.155215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34739 (local address 127.28.177.131:41021)
0808 02:17:40.155382 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:40.155425 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:40.155904 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:40.156258 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:40.156262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:40.156918 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:40.157132 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:40.157138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:40.157879 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:40.157881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:40.158738 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:40.158742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:40.159254 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:40.159255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:40.159394 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:40.160007 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:40.160016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:40.161711 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:40.163617 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:40.163625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:40.163628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:40.163856 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:40.164096 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:40.164097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:40.164097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:40.164224 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:40.164502 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:40.164509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:40.164659 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 02:17:40.164777 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:17:40.164963 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260808 02:17:40.165716 986 heartbeater.cc:347] Connected to a master server at 127.28.177.190:34739
I20260808 02:17:40.165829 986 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:40.166015 986 heartbeater.cc:511] Master 127.28.177.190:34739 requested a full tablet report, sending...
I20260808 02:17:40.166482 353 ts_manager.cc:194] Registered new tserver with Master: dcab6828010c4dee9e09eb9e740bf30f (127.28.177.131:44273)
I20260808 02:17:40.166829 353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:41021
I20260808 02:17:40.174373 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:17:40.190325 403 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:40.189179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.28.177.190:34739)
0808 02:17:40.189275 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:40.189277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:40.189367 (+ 90us) server_negotiation.cc:407] Connection header received
0808 02:17:40.189407 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:40.189408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:40.189423 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:40.189539 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:40.190257 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53818: BlockingRecv error: recv got EOF from 127.0.0.1:53818 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:17:40.191313 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 431
I20260808 02:17:40.212604 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 615
I20260808 02:17:40.224366 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 808
I20260808 02:17:40.235364 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 306
2026-08-08T02:17:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2233 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:40 dist-test-slave-3h5h krb5kdc[1004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:40 dist-test-slave-3h5h krb5kdc[1004](info): set up 2 sockets
Aug 08 02:17:40 dist-test-slave-3h5h krb5kdc[1004](info): commencing operation
krb5kdc: starting...
W20260808 02:17:41.582365 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:41 dist-test-slave-3h5h krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:41.717010 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41017
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:41.803560 1020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:41.803840 1020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:41.803877 1020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:41.805116 1020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:41.805192 1020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:41.805214 1020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:41.805236 1020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:41.805269 1020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:41.806834 1020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:41.807130 1020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:41.807370 1020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:41.812877 1025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:41.812872 1030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:41.813097 1020 server_base.cc:1034] running on GCE node
W20260808 02:17:41.812872 1026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:41.813439 1020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:41.813719 1020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:41.814903 1020 hybrid_clock.cc:648] HybridClock initialized: now 1786155461814894 us; error 36 us; skew 500 ppm
Aug 08 02:17:41 dist-test-slave-3h5h krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155461, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:41.821844 1020 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:41.822337 1020 webserver.cc:460] Webserver started at http://127.28.177.190:37949/ using document root <none> and password file <none>
I20260808 02:17:41.822557 1020 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:41.822599 1020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:41.822702 1020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:41.823837 1020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "a79e1b3d44ca438e98d19b7fc472bbed"
format_stamp: "Formatted at 2026-08-08 02:17:41 on dist-test-slave-3h5h"
server_key: "9653d311198b4faca8e375b7d0febe43"
server_key_iv: "f80fa94ebd6e2c9afa9e0910fdb04b79"
server_key_version: "encryptionkey@0"
I20260808 02:17:41.824157 1020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "a79e1b3d44ca438e98d19b7fc472bbed"
format_stamp: "Formatted at 2026-08-08 02:17:41 on dist-test-slave-3h5h"
server_key: "9653d311198b4faca8e375b7d0febe43"
server_key_iv: "f80fa94ebd6e2c9afa9e0910fdb04b79"
server_key_version: "encryptionkey@0"
I20260808 02:17:41.825734 1020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:41.826570 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:41.826814 1020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:41.826897 1020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "a79e1b3d44ca438e98d19b7fc472bbed"
format_stamp: "Formatted at 2026-08-08 02:17:41 on dist-test-slave-3h5h"
server_key: "9653d311198b4faca8e375b7d0febe43"
server_key_iv: "f80fa94ebd6e2c9afa9e0910fdb04b79"
server_key_version: "encryptionkey@0"
I20260808 02:17:41.826954 1020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:41.840761 1020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:41.843559 1020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:41.843703 1020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:41.848354 1020 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:41017
I20260808 02:17:41.848433 1106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:41017 every 8 connection(s)
I20260808 02:17:41.849040 1020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:41.849828 1108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:41.852295 1108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed: Bootstrap starting.
I20260808 02:17:41.852988 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 1020
I20260808 02:17:41.853062 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:41.853065 1108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:41.853296 29382 external_mini_cluster.cc:1428] Setting key bc79f93b33a1658682c95f9dfad49469
I20260808 02:17:41.853446 1108 log.cc:824] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:41.854422 1108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed: No bootstrap required, opened a new log
I20260808 02:17:41.857060 1108 raft_consensus.cc:348] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41017 } }
I20260808 02:17:41.857261 1108 raft_consensus.cc:374] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:41.857303 1108 raft_consensus.cc:729] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79e1b3d44ca438e98d19b7fc472bbed, State: Initialized, Role: FOLLOWER
I20260808 02:17:41.857501 1108 consensus_queue.cc:260] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41017 } }
I20260808 02:17:41.857609 1108 raft_consensus.cc:388] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:41.857659 1108 raft_consensus.cc:482] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:41.857715 1108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:41.858377 1108 raft_consensus.cc:504] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41017 } }
I20260808 02:17:41.858512 1108 leader_election.cc:302] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a79e1b3d44ca438e98d19b7fc472bbed; no voters:
I20260808 02:17:41.858768 1108 leader_election.cc:288] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:41.858832 1113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:41.859078 1108 sys_catalog.cc:564] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:41.860236 1113 raft_consensus.cc:686] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [term 1 LEADER]: Becoming Leader. State: Replica: a79e1b3d44ca438e98d19b7fc472bbed, State: Running, Role: LEADER
I20260808 02:17:41.860361 1113 consensus_queue.cc:237] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41017 } }
Aug 08 02:17:41 dist-test-slave-3h5h krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:41.861402 1114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader a79e1b3d44ca438e98d19b7fc472bbed. Latest consensus state: current_term: 1 leader_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41017 } } }
I20260808 02:17:41.861481 1114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:41.861124 1116 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79e1b3d44ca438e98d19b7fc472bbed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41017 } } }
W20260808 02:17:41.861629 1132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 02:17:41.861666 1116 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed [sys.catalog]: This master's current role is: LEADER
W20260808 02:17:41.861682 1132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:17:41.861799 1137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:41.861934 1137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:41.863652 1137 catalog_manager.cc:1347] Generated new cluster ID: 985468cc08b84720bf926df39c0326de
I20260808 02:17:41.863701 1137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:41.865522 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:41.853979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.28.177.190:41017)
0808 02:17:41.854165 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:41.854177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:41.854270 (+ 93us) server_negotiation.cc:407] Connection header received
0808 02:17:41.854857 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:41.854866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:41.854918 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:41.855070 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:41.855900 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:41.856827 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:41.857857 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:41.858298 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:41.858426 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:41.861710 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:41.861716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:41.861716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:41.861729 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:41.863816 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:41.864245 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:41.864248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:41.864248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:41.864340 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:41.864555 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:41.864556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:41.864556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:41.864875 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:41.865017 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:41.865163 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 02:17:41.865321 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260808 02:17:41.866484 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:17:41.886013 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:41.881197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.28.177.190:41017)
0808 02:17:41.881246 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:41.881248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:41.881371 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:17:41.881417 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:41.881417 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:41.881429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:41.881512 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:41.882004 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:41.882863 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:41.884103 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:41.884447 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:41.884529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:41.885425 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:41.885427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:41.885427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:41.885702 (+ 275us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:41.885724 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:41.885733 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:41.885799 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:41.885888 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:17:41.885919 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:17:41.893060 1137 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:41.893661 1137 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:41.897851 1137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a79e1b3d44ca438e98d19b7fc472bbed: Generated new TSK 0
I20260808 02:17:41.898053 1137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:17:41.903371 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 1020
2026-08-08T02:17:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1659 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:41 dist-test-slave-3h5h krb5kdc[1153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:41 dist-test-slave-3h5h krb5kdc[1153](info): set up 2 sockets
Aug 08 02:17:41 dist-test-slave-3h5h krb5kdc[1153](info): commencing operation
krb5kdc: starting...
W20260808 02:17:43.209939 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:43.353894 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42515
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:43.440130 1174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:43.440357 1174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:43.440380 1174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:43.441581 1174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:43.441625 1174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:43.441665 1174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:43.441679 1174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:43.441691 1174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:43.443151 1174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:43.443348 1174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:43.443536 1174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:43.449299 1183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:43.449298 1181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:43.449343 1180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:43.449579 1174 server_base.cc:1034] running on GCE node
I20260808 02:17:43.449944 1174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:43.450250 1174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:43.451465 1174 hybrid_clock.cc:648] HybridClock initialized: now 1786155463451411 us; error 66 us; skew 500 ppm
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:43.454584 1174 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:43.455114 1174 webserver.cc:460] Webserver started at http://127.28.177.190:40395/ using document root <none> and password file <none>
I20260808 02:17:43.455379 1174 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:43.455451 1174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:43.455602 1174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:43.456759 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "47ce0a4985b841228caff0ca35a851ef"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "7996937ffde46118bf2b4ee633b5d34e"
server_key_iv: "1fa353063c68b714e71d700ac31d3be0"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.457216 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "47ce0a4985b841228caff0ca35a851ef"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "7996937ffde46118bf2b4ee633b5d34e"
server_key_iv: "1fa353063c68b714e71d700ac31d3be0"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.458629 1174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:43.459484 1190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.459712 1174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:43.459859 1174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "47ce0a4985b841228caff0ca35a851ef"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "7996937ffde46118bf2b4ee633b5d34e"
server_key_iv: "1fa353063c68b714e71d700ac31d3be0"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.459944 1174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:43.488778 1174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:43.491433 1174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:43.491688 1174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:43.496663 1174 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42515
I20260808 02:17:43.496744 1266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42515 every 8 connection(s)
I20260808 02:17:43.497226 1174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:43.498565 1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:43.500734 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 1174
I20260808 02:17:43.500890 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:43.501122 29382 external_mini_cluster.cc:1428] Setting key 53bcb955d7ce4b32950164cc199ff964
I20260808 02:17:43.501843 1267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef: Bootstrap starting.
I20260808 02:17:43.502563 1267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:43.502919 1267 log.cc:824] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:43.503768 1267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef: No bootstrap required, opened a new log
I20260808 02:17:43.505949 1267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ce0a4985b841228caff0ca35a851ef" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42515 } }
I20260808 02:17:43.506145 1267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:43.506187 1267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ce0a4985b841228caff0ca35a851ef, State: Initialized, Role: FOLLOWER
I20260808 02:17:43.506335 1267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ce0a4985b841228caff0ca35a851ef" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42515 } }
I20260808 02:17:43.506457 1267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:43.506546 1267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:43.506629 1267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:43.507296 1267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ce0a4985b841228caff0ca35a851ef" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42515 } }
I20260808 02:17:43.507418 1267 leader_election.cc:302] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47ce0a4985b841228caff0ca35a851ef; no voters:
I20260808 02:17:43.507656 1267 leader_election.cc:288] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:43.507702 1273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:43.507850 1273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [term 1 LEADER]: Becoming Leader. State: Replica: 47ce0a4985b841228caff0ca35a851ef, State: Running, Role: LEADER
I20260808 02:17:43.507956 1273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ce0a4985b841228caff0ca35a851ef" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42515 } }
I20260808 02:17:43.507998 1267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:43.508306 1273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47ce0a4985b841228caff0ca35a851ef. Latest consensus state: current_term: 1 leader_uuid: "47ce0a4985b841228caff0ca35a851ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ce0a4985b841228caff0ca35a851ef" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42515 } } }
I20260808 02:17:43.508319 1274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47ce0a4985b841228caff0ca35a851ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ce0a4985b841228caff0ca35a851ef" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42515 } } }
I20260808 02:17:43.508390 1273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:43.508416 1274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:43.508802 1278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:43.509406 1278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:43.511157 1278 catalog_manager.cc:1347] Generated new cluster ID: d303efbf959d4c9baf9b5d19c74cd652
I20260808 02:17:43.511211 1278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:43.514017 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:43.501684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.28.177.190:42515)
0808 02:17:43.501894 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:43.501905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:43.501944 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:17:43.502471 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:43.502479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:43.502531 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:43.502699 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:43.503498 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.504436 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.505315 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.505726 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.505848 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:43.508773 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:43.508778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:43.508778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:43.508791 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:43.511486 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:43.511998 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:43.512002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:43.512002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:43.512126 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:43.512400 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:43.512401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:43.512402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:43.512772 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:43.512981 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:43.513588 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 02:17:43.513773 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260808 02:17:43.519857 1278 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:43.520489 1278 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:43.525797 1278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47ce0a4985b841228caff0ca35a851ef: Generated new TSK 0
I20260808 02:17:43.525992 1278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:43.569082 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42515
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:43.675637 1302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:43.675853 1302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:43.675879 1302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:43.677147 1302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:43.677191 1302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:43.677217 1302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:43.678705 1302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:43.678952 1302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:43.679216 1302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:43.684787 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:43.684788 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:43.684885 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:43.685151 1302 server_base.cc:1034] running on GCE node
I20260808 02:17:43.685364 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:43.685634 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:43.686800 1302 hybrid_clock.cc:648] HybridClock initialized: now 1786155463686777 us; error 46 us; skew 500 ppm
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:43.689502 1302 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:43.689999 1302 webserver.cc:460] Webserver started at http://127.28.177.129:32789/ using document root <none> and password file <none>
I20260808 02:17:43.690253 1302 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:43.690325 1302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:43.690483 1302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:43.691567 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "4b22b4e5185f46228065e93f635e561d"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "cf0279a4b31dd7b4082c86c4d46ee96d"
server_key_iv: "3c6d168fa08c4537927f12c2edab2498"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.691952 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "4b22b4e5185f46228065e93f635e561d"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "cf0279a4b31dd7b4082c86c4d46ee96d"
server_key_iv: "3c6d168fa08c4537927f12c2edab2498"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.693511 1302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:43.694417 1329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.694674 1302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:43.694809 1302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "4b22b4e5185f46228065e93f635e561d"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "cf0279a4b31dd7b4082c86c4d46ee96d"
server_key_iv: "3c6d168fa08c4537927f12c2edab2498"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.694905 1302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:43.715071 1302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:43.717734 1302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:43.717947 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:43.718333 1302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:43.718750 1302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:43.718816 1302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.718863 1302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:43.718895 1302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.725718 1302 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45559
I20260808 02:17:43.725770 1480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45559 every 8 connection(s)
I20260808 02:17:43.726148 1302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:43.726863 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 1302
I20260808 02:17:43.727005 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:43.727228 29382 external_mini_cluster.cc:1428] Setting key e528538e9937fd9e2206aceefe44c347
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:17:43.738641 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:43.726979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:44719 (local address 127.28.177.190:42515)
0808 02:17:43.727051 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:43.727054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:43.727337 (+ 283us) server_negotiation.cc:407] Connection header received
0808 02:17:43.728031 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:43.728032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:43.728045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:43.728122 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:43.729599 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.730323 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.731813 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.732213 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.732299 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:43.735004 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:43.735006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:43.735006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:43.735016 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:43.736988 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:43.737460 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:43.737462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:43.737462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:43.737537 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:43.737824 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:43.737825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:43.737826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:43.738003 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:43.738104 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:43.738415 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 02:17:43.738515 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:17:43.738677 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:43.726935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42515 (local address 127.28.177.129:44719)
0808 02:17:43.727198 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:43.727230 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:43.727915 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:43.728232 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:43.728237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:43.729178 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:43.729474 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:43.729480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.730433 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.730435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:43.731701 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:43.731704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.732321 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.732322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:43.732433 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:43.733049 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:43.733064 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:43.734900 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:43.737121 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:43.737128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:43.737130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:43.737372 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:43.737622 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:43.737623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:43.737623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:43.737744 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:43.738122 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:43.738129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:43.738264 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:17:43.738357 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:17:43.738487 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260808 02:17:43.739408 1482 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42515
I20260808 02:17:43.739512 1482 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:43.739760 1482 heartbeater.cc:511] Master 127.28.177.190:42515 requested a full tablet report, sending...
I20260808 02:17:43.740530 1212 ts_manager.cc:194] Registered new tserver with Master: 4b22b4e5185f46228065e93f635e561d (127.28.177.129:45559)
I20260808 02:17:43.741577 1212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:44719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:43.760392 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42515
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:43.851639 1488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:43.851922 1488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:43.851965 1488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:43.853274 1488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:43.853358 1488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:43.853405 1488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:43.855093 1488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:43.855440 1488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:43.855726 1488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:43.861699 1495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:43.861699 1497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:43.861747 1503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:43.862067 1488 server_base.cc:1034] running on GCE node
I20260808 02:17:43.862308 1488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:43.862586 1488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:43.863744 1488 hybrid_clock.cc:648] HybridClock initialized: now 1786155463863726 us; error 32 us; skew 500 ppm
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:43.866573 1488 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:43.867080 1488 webserver.cc:460] Webserver started at http://127.28.177.130:40159/ using document root <none> and password file <none>
I20260808 02:17:43.867359 1488 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:43.867431 1488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:43.867595 1488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:43.868706 1488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "c9d12335a69e42939ce16ab882e9791b"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "2099e92f71313faae6903ca3198cb848"
server_key_iv: "34a77b2983a1d0c168ab08ea25efa117"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.869165 1488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "c9d12335a69e42939ce16ab882e9791b"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "2099e92f71313faae6903ca3198cb848"
server_key_iv: "34a77b2983a1d0c168ab08ea25efa117"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.870635 1488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:43.871532 1511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.871786 1488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:43.871928 1488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "c9d12335a69e42939ce16ab882e9791b"
format_stamp: "Formatted at 2026-08-08 02:17:43 on dist-test-slave-3h5h"
server_key: "2099e92f71313faae6903ca3198cb848"
server_key_iv: "34a77b2983a1d0c168ab08ea25efa117"
server_key_version: "encryptionkey@0"
I20260808 02:17:43.872014 1488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:43.892017 1488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:43.894645 1488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:43.894846 1488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:43.895125 1488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:43.895507 1488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:43.895570 1488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.895612 1488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:43.895642 1488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:43.902406 1488 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:33409
I20260808 02:17:43.902454 1676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:33409 every 8 connection(s)
I20260808 02:17:43.902868 1488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:17:43.907970 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 1488
I20260808 02:17:43.908099 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:43.908365 29382 external_mini_cluster.cc:1428] Setting key 0ab3c3055b1b1580ccba168933a69262
Aug 08 02:17:43 dist-test-slave-3h5h krb5kdc[1153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:43.916002 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:43.903742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:46895 (local address 127.28.177.190:42515)
0808 02:17:43.903828 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:43.903830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:43.904045 (+ 215us) server_negotiation.cc:407] Connection header received
0808 02:17:43.904632 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:43.904634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:43.904648 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:43.904726 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:43.906249 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.907159 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.909072 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.909386 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.909461 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:43.912635 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:43.912638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:43.912638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:43.912648 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:43.914244 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:43.914798 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:43.914800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:43.914800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:43.914866 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:43.915195 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:43.915196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:43.915196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:43.915320 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:43.915380 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:43.915839 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 02:17:43.915924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:17:43.916316 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:43.903694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42515 (local address 127.28.177.130:46895)
0808 02:17:43.903877 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:43.903923 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:43.904485 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:43.904922 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:43.904930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:43.905768 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:43.906119 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:43.906127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.907301 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.907302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:43.908978 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:43.908982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:43.909494 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:43.909495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:43.909678 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:43.910333 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:43.910345 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:43.912530 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:43.914329 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:43.914338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:43.914340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:43.914694 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:43.914952 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:43.914953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:43.914953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:43.915113 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:43.915395 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:43.915402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:43.915605 (+ 203us) client_negotiation.cc:769] Sending connection context
0808 02:17:43.915748 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 02:17:43.915986 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260808 02:17:43.916939 1679 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42515
I20260808 02:17:43.917071 1679 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:43.917313 1679 heartbeater.cc:511] Master 127.28.177.190:42515 requested a full tablet report, sending...
I20260808 02:17:43.917819 1212 ts_manager.cc:194] Registered new tserver with Master: c9d12335a69e42939ce16ab882e9791b (127.28.177.130:33409)
I20260808 02:17:43.918252 1212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:46895
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:43.940428 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42515
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:44.030562 1688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:44.030776 1688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:44.030800 1688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:44.031922 1688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:44.031958 1688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:44.031988 1688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:44.033499 1688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:44.033721 1688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:44.033900 1688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:44.039469 1694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:44.039479 1688 server_base.cc:1034] running on GCE node
W20260808 02:17:44.039422 1693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:44.039422 1697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:44.040123 1688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:44.040429 1688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:44.041630 1688 hybrid_clock.cc:648] HybridClock initialized: now 1786155464041613 us; error 32 us; skew 500 ppm
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155464, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:44.044497 1688 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:44.045127 1688 webserver.cc:460] Webserver started at http://127.28.177.131:34183/ using document root <none> and password file <none>
I20260808 02:17:44.045378 1688 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:44.045464 1688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:44.045612 1688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:44.046689 1688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "9f5371aa6fb04f649ea62a0eb5fcb60a"
format_stamp: "Formatted at 2026-08-08 02:17:44 on dist-test-slave-3h5h"
server_key: "0236f2c53d38a490772805bf4436c5aa"
server_key_iv: "e26fcfd514f7e8fbba697861c7dd81e7"
server_key_version: "encryptionkey@0"
I20260808 02:17:44.047013 1688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "9f5371aa6fb04f649ea62a0eb5fcb60a"
format_stamp: "Formatted at 2026-08-08 02:17:44 on dist-test-slave-3h5h"
server_key: "0236f2c53d38a490772805bf4436c5aa"
server_key_iv: "e26fcfd514f7e8fbba697861c7dd81e7"
server_key_version: "encryptionkey@0"
I20260808 02:17:44.048424 1688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:44.049494 1710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:44.049736 1688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:44.049836 1688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "9f5371aa6fb04f649ea62a0eb5fcb60a"
format_stamp: "Formatted at 2026-08-08 02:17:44 on dist-test-slave-3h5h"
server_key: "0236f2c53d38a490772805bf4436c5aa"
server_key_iv: "e26fcfd514f7e8fbba697861c7dd81e7"
server_key_version: "encryptionkey@0"
I20260808 02:17:44.049901 1688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:44.069924 1688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:44.072556 1688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:44.072750 1688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:44.073835 1688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:44.074416 1688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:44.074553 1688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:44.074605 1688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:44.074636 1688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:44.081544 1688 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:40427
I20260808 02:17:44.081620 1882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:40427 every 8 connection(s)
I20260808 02:17:44.082006 1688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:17:44.087219 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 1688
I20260808 02:17:44.087321 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:44.087532 29382 external_mini_cluster.cc:1428] Setting key 281cd8ef17128eba5d022f956e1cef80
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155464, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:44.092796 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:44.082844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:50897 (local address 127.28.177.190:42515)
0808 02:17:44.082940 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:44.082942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:44.083172 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:17:44.083659 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:44.083661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:44.083673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:44.083744 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:44.084767 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:44.085390 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:44.086495 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:44.086722 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:44.086793 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:44.089730 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:44.089732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:44.089732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:44.089740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:44.091328 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:44.091802 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:44.091802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:44.091803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:44.091863 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:44.092177 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:44.092178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:44.092178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:44.092297 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:44.092361 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:44.092662 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 02:17:44.092729 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:17:44.093130 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:44.082801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42515 (local address 127.28.177.131:50897)
0808 02:17:44.082997 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:44.083044 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:44.083535 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:44.083798 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:44.083802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:44.084453 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:44.084649 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:44.084655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:44.085487 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:44.085488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:44.086363 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:44.086366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:44.086831 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:44.086833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:44.087080 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:44.087951 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:44.088030 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:44.089632 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:44.091418 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:44.091425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:44.091427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:44.091725 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:44.091933 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:44.091934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:44.091934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:44.092052 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:44.092361 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:44.092366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:44.092496 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 02:17:44.092637 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 02:17:44.092776 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260808 02:17:44.093770 1883 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42515
I20260808 02:17:44.093884 1883 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:44.094110 1883 heartbeater.cc:511] Master 127.28.177.190:42515 requested a full tablet report, sending...
I20260808 02:17:44.094548 1212 ts_manager.cc:194] Registered new tserver with Master: 9f5371aa6fb04f649ea62a0eb5fcb60a (127.28.177.131:40427)
I20260808 02:17:44.094926 1212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:50897
I20260808 02:17:44.099627 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:44.125979 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:44.117647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.28.177.190:42515)
0808 02:17:44.117731 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:44.117734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:44.117837 (+ 103us) server_negotiation.cc:407] Connection header received
0808 02:17:44.117915 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:44.117917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:44.117928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:44.117981 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:44.118851 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:44.119419 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:44.120270 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:44.120504 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:44.120612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:44.123292 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:44.123294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:44.123294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:44.123302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:44.124821 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:44.125236 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:44.125237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:44.125238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:44.125294 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:44.125479 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:44.125479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:44.125480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:44.125659 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:44.125740 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:44.125841 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 02:17:44.125910 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 02:17:44.126372 1212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47104
I20260808 02:17:44.126893 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 1302
I20260808 02:17:44.141919 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 1488
I20260808 02:17:44.151338 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 1688
I20260808 02:17:44.162882 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 1174
2026-08-08T02:17:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2262 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1900](info): set up 2 sockets
Aug 08 02:17:44 dist-test-slave-3h5h krb5kdc[1900](info): commencing operation
krb5kdc: starting...
W20260808 02:17:45.428687 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:45 dist-test-slave-3h5h krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:45.569085 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42593
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:45.655541 1918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:45.655762 1918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:45.655784 1918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:45.656932 1918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:45.656971 1918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:45.656985 1918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:45.656998 1918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:45.657011 1918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:45.658505 1918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:45.658715 1918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:45.658897 1918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:45.664062 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:45.664108 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:45.664196 1928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:45.664217 1918 server_base.cc:1034] running on GCE node
I20260808 02:17:45.664549 1918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:45.664929 1918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:45.666100 1918 hybrid_clock.cc:648] HybridClock initialized: now 1786155465666074 us; error 46 us; skew 500 ppm
Aug 08 02:17:45 dist-test-slave-3h5h krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155465, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:45.668756 1918 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:45.669301 1918 webserver.cc:460] Webserver started at http://127.28.177.190:40313/ using document root <none> and password file <none>
I20260808 02:17:45.669515 1918 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:45.669559 1918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:45.669656 1918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:45.670696 1918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "dbb1671bd2534b9296e3df616df833ab"
format_stamp: "Formatted at 2026-08-08 02:17:45 on dist-test-slave-3h5h"
server_key: "c17d00b834a7c443d231c51de55459e8"
server_key_iv: "89001566cddf298f9fe2547cf53b293c"
server_key_version: "encryptionkey@0"
I20260808 02:17:45.671012 1918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "dbb1671bd2534b9296e3df616df833ab"
format_stamp: "Formatted at 2026-08-08 02:17:45 on dist-test-slave-3h5h"
server_key: "c17d00b834a7c443d231c51de55459e8"
server_key_iv: "89001566cddf298f9fe2547cf53b293c"
server_key_version: "encryptionkey@0"
I20260808 02:17:45.672340 1918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:17:45.673163 1945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:45.673343 1918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:45.673457 1918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "dbb1671bd2534b9296e3df616df833ab"
format_stamp: "Formatted at 2026-08-08 02:17:45 on dist-test-slave-3h5h"
server_key: "c17d00b834a7c443d231c51de55459e8"
server_key_iv: "89001566cddf298f9fe2547cf53b293c"
server_key_version: "encryptionkey@0"
I20260808 02:17:45.673518 1918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:45.683408 1918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:45.685660 1918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:45.685781 1918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:45.689882 1918 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42593
I20260808 02:17:45.690246 2029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42593 every 8 connection(s)
I20260808 02:17:45.690280 1918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:45.691252 2031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:45.694101 2031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab: Bootstrap starting.
I20260808 02:17:45.694800 2031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:45.695142 2031 log.cc:824] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:45.695683 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 1918
I20260808 02:17:45.695804 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:45.696033 29382 external_mini_cluster.cc:1428] Setting key eb572a921e8dee69f81bef37cf7e73c2
I20260808 02:17:45.696192 2031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab: No bootstrap required, opened a new log
I20260808 02:17:45.698799 2031 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb1671bd2534b9296e3df616df833ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42593 } }
I20260808 02:17:45.698990 2031 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:45.699044 2031 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb1671bd2534b9296e3df616df833ab, State: Initialized, Role: FOLLOWER
I20260808 02:17:45.699230 2031 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb1671bd2534b9296e3df616df833ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42593 } }
I20260808 02:17:45.699328 2031 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:45.699378 2031 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:45.699438 2031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:45.700073 2031 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb1671bd2534b9296e3df616df833ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42593 } }
I20260808 02:17:45.700207 2031 leader_election.cc:302] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbb1671bd2534b9296e3df616df833ab; no voters:
I20260808 02:17:45.700440 2031 leader_election.cc:288] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:45.700492 2037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:45.700659 2037 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [term 1 LEADER]: Becoming Leader. State: Replica: dbb1671bd2534b9296e3df616df833ab, State: Running, Role: LEADER
I20260808 02:17:45.700765 2031 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:45.700820 2037 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb1671bd2534b9296e3df616df833ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42593 } }
I20260808 02:17:45.701310 2039 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb1671bd2534b9296e3df616df833ab. Latest consensus state: current_term: 1 leader_uuid: "dbb1671bd2534b9296e3df616df833ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb1671bd2534b9296e3df616df833ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42593 } } }
I20260808 02:17:45.701359 2038 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbb1671bd2534b9296e3df616df833ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb1671bd2534b9296e3df616df833ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42593 } } }
I20260808 02:17:45.701522 2038 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:45.701421 2039 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:45.701831 2047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:45.702522 2047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:45.704310 2047 catalog_manager.cc:1347] Generated new cluster ID: 5df255e119fc419abe500ed4fe560dcd
I20260808 02:17:45.704372 2047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:17:45 dist-test-slave-3h5h krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:45.709322 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:45.696651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.28.177.190:42593)
0808 02:17:45.696898 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:45.696907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:45.696944 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:17:45.698788 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:45.698798 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:45.698857 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:45.699067 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:45.699917 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:45.701394 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:45.702548 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:45.703148 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:45.703274 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:45.705880 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:45.705885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:45.705885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:45.705898 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:45.707601 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:45.708014 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:45.708018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:45.708018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:45.708125 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:45.708346 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:45.708347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:45.708347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:45.708596 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:45.708788 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:45.708890 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 02:17:45.709152 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":112,"threads_started":1}
I20260808 02:17:45.741708 2047 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:45.742491 2047 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:45.746974 2047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb1671bd2534b9296e3df616df833ab: Generated new TSK 0
I20260808 02:17:45.747192 2047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:45.780449 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42593
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:45.870143 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:45.870398 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:45.870441 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:45.871662 2065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:45.871737 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:45.871780 2065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:45.873539 2065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:45.873852 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:45.874102 2065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:45.879702 2076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:45.879671 2073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:45.879671 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:45.879878 2065 server_base.cc:1034] running on GCE node
I20260808 02:17:45.880128 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:45.880367 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:45.881537 2065 hybrid_clock.cc:648] HybridClock initialized: now 1786155465881529 us; error 38 us; skew 500 ppm
Aug 08 02:17:45 dist-test-slave-3h5h krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155465, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:45.884093 2065 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:45.884543 2065 webserver.cc:460] Webserver started at http://127.28.177.129:40953/ using document root <none> and password file <none>
I20260808 02:17:45.884824 2065 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:45.884964 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:45.885123 2065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:45.886210 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "6f88cfc3f5f74163b8d73a2fa72d3ca1"
format_stamp: "Formatted at 2026-08-08 02:17:45 on dist-test-slave-3h5h"
server_key: "9e0cf46950776315c960d4474d76bc8a"
server_key_iv: "d0c5b7b0e36b53f306eb81a8e566dc02"
server_key_version: "encryptionkey@0"
I20260808 02:17:45.886582 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "6f88cfc3f5f74163b8d73a2fa72d3ca1"
format_stamp: "Formatted at 2026-08-08 02:17:45 on dist-test-slave-3h5h"
server_key: "9e0cf46950776315c960d4474d76bc8a"
server_key_iv: "d0c5b7b0e36b53f306eb81a8e566dc02"
server_key_version: "encryptionkey@0"
I20260808 02:17:45.887986 2065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:45.888804 2086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:45.889011 2065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:45.889117 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "6f88cfc3f5f74163b8d73a2fa72d3ca1"
format_stamp: "Formatted at 2026-08-08 02:17:45 on dist-test-slave-3h5h"
server_key: "9e0cf46950776315c960d4474d76bc8a"
server_key_iv: "d0c5b7b0e36b53f306eb81a8e566dc02"
server_key_version: "encryptionkey@0"
I20260808 02:17:45.889204 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:45.914893 2065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:45.917271 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:45.917454 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:45.917723 2065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:45.918071 2065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:45.918138 2065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:45.918185 2065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:45.918210 2065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:45.924216 2065 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:33595
I20260808 02:17:45.924270 2251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:33595 every 8 connection(s)
I20260808 02:17:45.924649 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:45.927583 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 2065
I20260808 02:17:45.927692 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:45.927907 29382 external_mini_cluster.cc:1428] Setting key b426de437a5d493fe34afe6d675c96a0
Aug 08 02:17:45 dist-test-slave-3h5h krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155465, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:45.935317 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:45.925543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:57619 (local address 127.28.177.190:42593)
0808 02:17:45.925611 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:45.925613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:45.925828 (+ 215us) server_negotiation.cc:407] Connection header received
0808 02:17:45.926930 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:45.926931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:45.926948 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:45.927002 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:45.927802 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:45.928390 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:45.929498 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:45.929730 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:45.929820 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:45.932272 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:45.932274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:45.932274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:45.932282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:45.933847 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:45.934279 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:45.934281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:45.934281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:45.934346 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:45.934605 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:45.934607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:45.934607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:45.934748 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:45.934831 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:45.935114 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 02:17:45.935210 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:17:45.935549 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:45.925506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42593 (local address 127.28.177.129:57619)
0808 02:17:45.925703 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:45.925740 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:45.926797 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:45.927090 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:45.927093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:45.927389 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:45.927691 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:45.927698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:45.928495 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:45.928496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:45.929399 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:45.929401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:45.929833 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:45.929834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:45.929959 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:45.930505 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:45.930514 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:45.932164 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:45.933947 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:45.933956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:45.933958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:45.934198 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:45.934416 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:45.934417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:45.934417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:45.934534 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:45.934822 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:45.934829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:45.934976 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 02:17:45.935089 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:17:45.935302 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260808 02:17:45.936095 2253 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42593
I20260808 02:17:45.936208 2253 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:45.936434 2253 heartbeater.cc:511] Master 127.28.177.190:42593 requested a full tablet report, sending...
I20260808 02:17:45.937101 1969 ts_manager.cc:194] Registered new tserver with Master: 6f88cfc3f5f74163b8d73a2fa72d3ca1 (127.28.177.129:33595)
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:17:45.937896 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:57619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:45.959781 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42593
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:46.044862 2262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:46.045166 2262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:46.045208 2262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:46.046406 2262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:46.046481 2262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:46.046533 2262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:46.048118 2262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:46.048408 2262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:46.048664 2262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:46.054018 2274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:46.054045 2272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:46.054018 2270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:46.054643 2262 server_base.cc:1034] running on GCE node
I20260808 02:17:46.054901 2262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:46.055172 2262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:46.056428 2262 hybrid_clock.cc:648] HybridClock initialized: now 1786155466056406 us; error 37 us; skew 500 ppm
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:46.059192 2262 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:46.059684 2262 webserver.cc:460] Webserver started at http://127.28.177.130:33883/ using document root <none> and password file <none>
I20260808 02:17:46.059959 2262 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:46.060034 2262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:46.060190 2262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:46.061373 2262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "8f4d40dd2aa04c298404d0fb9db19742"
format_stamp: "Formatted at 2026-08-08 02:17:46 on dist-test-slave-3h5h"
server_key: "d348142e33e82f5961fe70c3e575f86f"
server_key_iv: "4bb032a1d06672823050d2ab119e4186"
server_key_version: "encryptionkey@0"
I20260808 02:17:46.061758 2262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "8f4d40dd2aa04c298404d0fb9db19742"
format_stamp: "Formatted at 2026-08-08 02:17:46 on dist-test-slave-3h5h"
server_key: "d348142e33e82f5961fe70c3e575f86f"
server_key_iv: "4bb032a1d06672823050d2ab119e4186"
server_key_version: "encryptionkey@0"
I20260808 02:17:46.063230 2262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:46.064070 2286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:46.064273 2262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:46.064389 2262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "8f4d40dd2aa04c298404d0fb9db19742"
format_stamp: "Formatted at 2026-08-08 02:17:46 on dist-test-slave-3h5h"
server_key: "d348142e33e82f5961fe70c3e575f86f"
server_key_iv: "4bb032a1d06672823050d2ab119e4186"
server_key_version: "encryptionkey@0"
I20260808 02:17:46.064476 2262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:46.085098 2262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:46.087496 2262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:46.087702 2262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:46.088001 2262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:46.088382 2262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:46.088446 2262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:46.088498 2262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:46.088553 2262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:46.095709 2262 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:33395
I20260808 02:17:46.095754 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:33395 every 8 connection(s)
I20260808 02:17:46.096212 2262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:46.106383 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 2262
I20260808 02:17:46.106523 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:46.106755 29382 external_mini_cluster.cc:1428] Setting key f9623e0419c205734bd45ae9cf5fd245
I20260808 02:17:46.107208 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:46.097151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:44649 (local address 127.28.177.190:42593)
0808 02:17:46.097246 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:46.097249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:46.097421 (+ 172us) server_negotiation.cc:407] Connection header received
0808 02:17:46.098424 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:46.098426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:46.098439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:46.098524 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:46.099181 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.099743 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.100942 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.101246 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.101413 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:46.103848 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:46.103850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:46.103850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:46.103860 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:46.105560 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.105968 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.105969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.105969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.106106 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.106391 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.106392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.106393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.106571 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:46.106651 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:46.106991 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 02:17:46.107117 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:17:46.107437 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:46.097077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42593 (local address 127.28.177.130:44649)
0808 02:17:46.097261 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:46.097306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:46.098310 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:46.098581 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:46.098585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:46.098846 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:46.099046 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:46.099051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.099835 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.099836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:46.100789 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:46.100792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.101429 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.101430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:46.101534 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:46.102181 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:46.102190 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:46.103721 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:46.105644 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:46.105654 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:46.105657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:46.105896 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:46.106214 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:46.106215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:46.106215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:46.106321 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:46.106663 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:46.106670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:46.106830 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 02:17:46.106956 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 02:17:46.107187 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260808 02:17:46.107944 2458 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42593
I20260808 02:17:46.108047 2458 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:46.108285 2458 heartbeater.cc:511] Master 127.28.177.190:42593 requested a full tablet report, sending...
I20260808 02:17:46.108765 1969 ts_manager.cc:194] Registered new tserver with Master: 8f4d40dd2aa04c298404d0fb9db19742 (127.28.177.130:33395)
I20260808 02:17:46.109331 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:44649
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:46.136170 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42593
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:46.220928 2464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:46.221139 2464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:46.221163 2464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:46.222334 2464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:46.222378 2464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:46.222404 2464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:46.223838 2464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:46.224016 2464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:46.224188 2464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:46.229611 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:46.229607 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:46.229629 2464 server_base.cc:1034] running on GCE node
W20260808 02:17:46.229607 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:46.230298 2464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:46.230576 2464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:46.231783 2464 hybrid_clock.cc:648] HybridClock initialized: now 1786155466231769 us; error 35 us; skew 500 ppm
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:46.234755 2464 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:46.235335 2464 webserver.cc:460] Webserver started at http://127.28.177.131:41093/ using document root <none> and password file <none>
I20260808 02:17:46.235631 2464 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:46.235709 2464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:46.235878 2464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:46.237174 2464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "dd701d796f1541b69a71a1794c61e25a"
format_stamp: "Formatted at 2026-08-08 02:17:46 on dist-test-slave-3h5h"
server_key: "475aaa549638330c655a6c8a54f020e7"
server_key_iv: "ff630a55ad5bafda318ddb47b8996073"
server_key_version: "encryptionkey@0"
I20260808 02:17:46.237582 2464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "dd701d796f1541b69a71a1794c61e25a"
format_stamp: "Formatted at 2026-08-08 02:17:46 on dist-test-slave-3h5h"
server_key: "475aaa549638330c655a6c8a54f020e7"
server_key_iv: "ff630a55ad5bafda318ddb47b8996073"
server_key_version: "encryptionkey@0"
I20260808 02:17:46.239114 2464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:46.239941 2481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:46.240159 2464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:46.240284 2464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "dd701d796f1541b69a71a1794c61e25a"
format_stamp: "Formatted at 2026-08-08 02:17:46 on dist-test-slave-3h5h"
server_key: "475aaa549638330c655a6c8a54f020e7"
server_key_iv: "ff630a55ad5bafda318ddb47b8996073"
server_key_version: "encryptionkey@0"
I20260808 02:17:46.240365 2464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:46.257340 2464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:46.259889 2464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:46.260151 2464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:46.260443 2464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:46.260885 2464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:46.260948 2464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:46.260998 2464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:46.261056 2464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:46.267939 2464 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:33665
I20260808 02:17:46.267982 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:33665 every 8 connection(s)
I20260808 02:17:46.268404 2464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:17:46.272168 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 2464
I20260808 02:17:46.272258 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:46.272527 29382 external_mini_cluster.cc:1428] Setting key 6d70807ebc1219264f7046a07eda0acd
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:46.279534 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:46.269289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:38501 (local address 127.28.177.190:42593)
0808 02:17:46.269340 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:46.269343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:46.269537 (+ 194us) server_negotiation.cc:407] Connection header received
0808 02:17:46.270602 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:46.270603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:46.270618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:46.270686 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:46.271397 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.271900 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.273422 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.273775 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.273841 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:46.276373 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:46.276375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:46.276375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:46.276382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:46.278061 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.278505 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.278506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.278506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.278568 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.278821 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.278822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.278822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.278947 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:46.279026 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:46.279391 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 02:17:46.279464 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:46.279698 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:46.269247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42593 (local address 127.28.177.131:38501)
0808 02:17:46.269398 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:46.269446 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:46.270479 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:46.270761 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:46.270766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:46.271033 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:46.271290 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:46.271295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.271998 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.271999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:46.273342 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:46.273345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.273859 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.273860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:46.273989 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:46.274556 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:46.274565 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:46.276260 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:46.278177 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:46.278184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:46.278186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:46.278419 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:46.278641 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:46.278642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:46.278642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:46.278755 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:46.279083 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:46.279089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:46.279216 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 02:17:46.279331 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:17:46.279461 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260808 02:17:46.280256 2648 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42593
I20260808 02:17:46.280377 2648 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:46.280572 2648 heartbeater.cc:511] Master 127.28.177.190:42593 requested a full tablet report, sending...
I20260808 02:17:46.281037 1969 ts_manager.cc:194] Registered new tserver with Master: dd701d796f1541b69a71a1794c61e25a (127.28.177.131:33665)
I20260808 02:17:46.281440 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:38501
I20260808 02:17:46.284507 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:46.312420 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:46.303509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.28.177.129:33595)
0808 02:17:46.303719 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:46.303725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:46.303736 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:46.303767 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:46.303769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:46.303785 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:46.303862 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:46.304579 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.305582 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.306440 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.306804 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.306871 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:46.309508 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:46.309518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:46.309519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:46.309526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:46.311189 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.311584 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.311588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.311588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.311660 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.311877 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.311878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.311878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.312037 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:46.312211 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:46.312259 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 02:17:46.312332 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":116,"threads_started":1}
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:46.322391 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:46.314441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.28.177.190:42593)
0808 02:17:46.314547 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:46.314549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:46.314558 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:46.314735 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:46.314736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:46.314749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:46.314812 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:46.315355 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.315878 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.316775 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:46.317049 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:46.317113 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:46.319627 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:46.319629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:46.319629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:46.319646 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:46.321056 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.321417 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.321419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.321419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.321499 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:46.321724 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:46.321725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:46.321725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:46.321898 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:46.322008 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:46.322177 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:17:46.322266 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 02:17:46.322659 1969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102
I20260808 02:17:46.323191 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 2065
I20260808 02:17:46.340072 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 2262
I20260808 02:17:46.350939 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 2464
I20260808 02:17:46.361209 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 1918
2026-08-08T02:17:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2197 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[2671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[2671](info): set up 2 sockets
Aug 08 02:17:46 dist-test-slave-3h5h krb5kdc[2671](info): commencing operation
krb5kdc: starting...
W20260808 02:17:47.614878 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:47 dist-test-slave-3h5h krb5kdc[2671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:47.753391 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:38103
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:38103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:47.848083 2693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:47.848338 2693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:47.848366 2693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:47.849973 2693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:47.850031 2693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:47.850049 2693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:47.850098 2693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:47.850113 2693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:47.852007 2693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:38103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:47.852341 2693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:47.852617 2693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:47.858556 2704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:47.858621 2693 server_base.cc:1034] running on GCE node
W20260808 02:17:47.858549 2700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:47.858549 2699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:47.859046 2693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:47.859397 2693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:47.860579 2693 hybrid_clock.cc:648] HybridClock initialized: now 1786155467860545 us; error 56 us; skew 500 ppm
Aug 08 02:17:47 dist-test-slave-3h5h krb5kdc[2671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155467, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:47.868427 2693 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:47.868958 2693 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:47.869037 2693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:47.869180 2693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:47.870328 2693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "496a0e9a29c24a1d8b6b052cc7df154c"
format_stamp: "Formatted at 2026-08-08 02:17:47 on dist-test-slave-3h5h"
server_key: "d2852e2896984e1b7e43916055799a7d"
server_key_iv: "2221670eef2380fd37e3512f6d8162ba"
server_key_version: "encryptionkey@0"
I20260808 02:17:47.870769 2693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "496a0e9a29c24a1d8b6b052cc7df154c"
format_stamp: "Formatted at 2026-08-08 02:17:47 on dist-test-slave-3h5h"
server_key: "d2852e2896984e1b7e43916055799a7d"
server_key_iv: "2221670eef2380fd37e3512f6d8162ba"
server_key_version: "encryptionkey@0"
I20260808 02:17:47.872141 2693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:47.872999 2716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:47.873183 2693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:47.873286 2693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "496a0e9a29c24a1d8b6b052cc7df154c"
format_stamp: "Formatted at 2026-08-08 02:17:47 on dist-test-slave-3h5h"
server_key: "d2852e2896984e1b7e43916055799a7d"
server_key_iv: "2221670eef2380fd37e3512f6d8162ba"
server_key_version: "encryptionkey@0"
I20260808 02:17:47.873365 2693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:47.885707 2693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:47.887779 2693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:47.887950 2693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:47.892516 2693 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:38103
I20260808 02:17:47.892563 2792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:38103 every 8 connection(s)
I20260808 02:17:47.892951 2693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:47.894454 2794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:47.897094 2794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c: Bootstrap starting.
I20260808 02:17:47.897727 2794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:47.898007 2794 log.cc:824] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:47.898803 2794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c: No bootstrap required, opened a new log
I20260808 02:17:47.900467 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 2693
I20260808 02:17:47.900611 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:47.900827 29382 external_mini_cluster.cc:1428] Setting key f8af0402bcb264315469bb4a7f53b057
I20260808 02:17:47.900897 2794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38103 } }
I20260808 02:17:47.901109 2794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:47.901193 2794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496a0e9a29c24a1d8b6b052cc7df154c, State: Initialized, Role: FOLLOWER
I20260808 02:17:47.901357 2794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38103 } }
I20260808 02:17:47.901453 2794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:47.901495 2794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:47.901590 2794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:47.902467 2794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38103 } }
I20260808 02:17:47.902601 2794 leader_election.cc:302] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 496a0e9a29c24a1d8b6b052cc7df154c; no voters:
I20260808 02:17:47.902788 2794 leader_election.cc:288] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:47.902856 2800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:47.903136 2794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:47.903136 2800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [term 1 LEADER]: Becoming Leader. State: Replica: 496a0e9a29c24a1d8b6b052cc7df154c, State: Running, Role: LEADER
I20260808 02:17:47.903340 2800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38103 } }
I20260808 02:17:47.903781 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 496a0e9a29c24a1d8b6b052cc7df154c. Latest consensus state: current_term: 1 leader_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38103 } } }
I20260808 02:17:47.903946 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:47.904210 2802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496a0e9a29c24a1d8b6b052cc7df154c" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38103 } } }
I20260808 02:17:47.904521 2802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:47.904703 2807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:47.905256 2807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:47.906999 2807 catalog_manager.cc:1347] Generated new cluster ID: 50bb73963f894b879c6371e8d2c6773e
I20260808 02:17:47.907068 2807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:17:47 dist-test-slave-3h5h krb5kdc[2671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:47.917102 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:47.901423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.28.177.190:38103)
0808 02:17:47.901654 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:47.901665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:47.901734 (+ 69us) server_negotiation.cc:407] Connection header received
0808 02:17:47.902271 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:47.902282 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:47.902356 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:47.902549 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:47.903508 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:47.905262 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:47.906534 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:47.909112 (+ 2578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:47.909257 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:47.912693 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:47.912699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:47.912699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:47.912715 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:47.915058 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:47.915554 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:47.915557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:47.915557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:47.915691 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:47.915955 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:47.915957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:47.915957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:47.916294 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:47.916453 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:47.916645 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:17:47.916808 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":105,"threads_started":1}
I20260808 02:17:47.920089 2807 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:47.920715 2807 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:47.925407 2807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c: Generated new TSK 0
I20260808 02:17:47.925666 2807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:47.960543 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38103
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:48.050992 2833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:48.051266 2833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:48.051308 2833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:48.052505 2833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:48.052577 2833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:48.052621 2833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:48.054216 2833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:48.054536 2833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:48.054780 2833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:48.060678 2840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:48.060673 2843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:48.060732 2841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:48.060712 2833 server_base.cc:1034] running on GCE node
I20260808 02:17:48.061388 2833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:48.061659 2833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:48.062819 2833 hybrid_clock.cc:648] HybridClock initialized: now 1786155468062808 us; error 34 us; skew 500 ppm
Aug 08 02:17:48 dist-test-slave-3h5h krb5kdc[2671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155468, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:48.065707 2833 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:48.066152 2833 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:48.066237 2833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:48.066406 2833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:48.067674 2833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "903a8b29ff8c40a596bd569894d4c68e"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "2ff1de3848a8d097a5ac178ad1fbb514"
server_key_iv: "2fa0f57e0009af8eaa63126774e65d21"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.068082 2833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "903a8b29ff8c40a596bd569894d4c68e"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "2ff1de3848a8d097a5ac178ad1fbb514"
server_key_iv: "2fa0f57e0009af8eaa63126774e65d21"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.069639 2833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:48.070494 2853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.070704 2833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:48.070820 2833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "903a8b29ff8c40a596bd569894d4c68e"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "2ff1de3848a8d097a5ac178ad1fbb514"
server_key_iv: "2fa0f57e0009af8eaa63126774e65d21"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.070906 2833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:48.090725 2833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:48.093041 2833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:48.093214 2833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:48.093513 2833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:48.093917 2833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:48.093982 2833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.094050 2833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:48.094084 2833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.101046 2833 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45921
I20260808 02:17:48.101203 3018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45921 every 8 connection(s)
I20260808 02:17:48.101456 2833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:48.107705 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 2833
I20260808 02:17:48.107870 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:48.108119 29382 external_mini_cluster.cc:1428] Setting key 05dbf4126282fabd8f863da0fbd19f3e
Aug 08 02:17:48 dist-test-slave-3h5h krb5kdc[2671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155468, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:48.113520 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.102339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:33927 (local address 127.28.177.190:38103)
0808 02:17:48.102418 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.102421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.102661 (+ 240us) server_negotiation.cc:407] Connection header received
0808 02:17:48.103252 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.103254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.103268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.103334 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:48.104706 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.105395 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.106582 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.106792 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.106861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.109749 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:48.109751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:48.109751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:48.109762 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:48.111470 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.112115 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.112117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.112117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.112185 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.112557 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.112558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.112559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.112734 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:48.112855 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.113263 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.113393 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:48.113592 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.102300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38103 (local address 127.28.177.129:33927)
0808 02:17:48.102518 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.102558 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.103127 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.103476 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.103481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.103778 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:48.104588 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.104596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.105595 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.105596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.106484 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.106486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.107035 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.107036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.107177 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.107866 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:48.107879 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:48.109651 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:48.111599 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:48.111608 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:48.111611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:48.112000 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:48.112308 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:48.112309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:48.112309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:48.112468 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:48.112903 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:48.112911 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:48.113098 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.113200 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.113339 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":125,"threads_started":1}
I20260808 02:17:48.114619 3020 heartbeater.cc:347] Connected to a master server at 127.28.177.190:38103
I20260808 02:17:48.114723 3020 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:48.114933 3020 heartbeater.cc:511] Master 127.28.177.190:38103 requested a full tablet report, sending...
I20260808 02:17:48.115662 2738 ts_manager.cc:194] Registered new tserver with Master: 903a8b29ff8c40a596bd569894d4c68e (127.28.177.129:45921)
I20260808 02:17:48.116366 2738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:33927
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:48.141434 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38103
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:48.231302 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:48.231559 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:48.231606 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:48.232781 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:48.232873 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:48.232931 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:48.234539 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:48.234843 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:48.235090 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:48.240638 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:48.240650 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:48.240653 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:48.240919 3025 server_base.cc:1034] running on GCE node
I20260808 02:17:48.241411 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:48.241736 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:48.242935 3025 hybrid_clock.cc:648] HybridClock initialized: now 1786155468242918 us; error 31 us; skew 500 ppm
Aug 08 02:17:48 dist-test-slave-3h5h krb5kdc[2671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155468, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:48.246112 3025 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:48.246532 3025 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:48.246619 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:48.246791 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:48.248054 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "9b06ff702e8a406b8183840f246e97c2"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "c2aa2e6b800dafa11cd0e788065dfe4d"
server_key_iv: "0e77205e44dc5ea81bc2ee28aaf708f7"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.248469 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "9b06ff702e8a406b8183840f246e97c2"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "c2aa2e6b800dafa11cd0e788065dfe4d"
server_key_iv: "0e77205e44dc5ea81bc2ee28aaf708f7"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.250377 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:17:48.251345 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.251633 3025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:48.251770 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "9b06ff702e8a406b8183840f246e97c2"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "c2aa2e6b800dafa11cd0e788065dfe4d"
server_key_iv: "0e77205e44dc5ea81bc2ee28aaf708f7"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.251860 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:48.269263 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:48.271689 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:48.271873 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:48.272146 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:48.272531 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:48.272599 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.272646 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:48.272680 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.279891 3025 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:39257
I20260808 02:17:48.279963 3217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:39257 every 8 connection(s)
I20260808 02:17:48.280336 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:17:48 dist-test-slave-3h5h krb5kdc[2671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155468, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:48.287984 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 3025
I20260808 02:17:48.288100 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:48.288329 29382 external_mini_cluster.cc:1428] Setting key e8800441aa27858b36facda22c77d467
I20260808 02:17:48.290515 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.281220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:39683 (local address 127.28.177.190:38103)
0808 02:17:48.281300 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.281302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.281479 (+ 177us) server_negotiation.cc:407] Connection header received
0808 02:17:48.281989 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.281990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.282001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.282078 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:48.282989 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.283670 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.284752 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.285103 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.285171 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.287370 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:48.287371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:48.287371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:48.287379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:48.289150 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.289529 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.289530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.289530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.289596 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.289803 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.289804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.289804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.289957 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:48.290037 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.290325 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.290424 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:17:48.290747 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.281174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38103 (local address 127.28.177.130:39683)
0808 02:17:48.281342 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.281385 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.281897 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.282135 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.282138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.282399 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:48.282902 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.282907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.283770 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.283771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.284646 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.284649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.285191 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.285192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.285297 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.285870 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:48.285880 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:48.287261 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:48.289225 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:48.289231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:48.289233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:48.289451 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:48.289658 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:48.289659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:48.289659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:48.289743 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:48.290025 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:48.290030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:48.290156 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.290285 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.290445 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260808 02:17:48.291348 3218 heartbeater.cc:347] Connected to a master server at 127.28.177.190:38103
I20260808 02:17:48.291462 3218 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:48.291632 3218 heartbeater.cc:511] Master 127.28.177.190:38103 requested a full tablet report, sending...
I20260808 02:17:48.292025 2738 ts_manager.cc:194] Registered new tserver with Master: 9b06ff702e8a406b8183840f246e97c2 (127.28.177.130:39257)
I20260808 02:17:48.292398 2738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:39683
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:48.320616 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38103
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:48.408104 3223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:48.408320 3223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:48.408347 3223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:48.409595 3223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:48.409644 3223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:48.409680 3223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:48.411252 3223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:48.411533 3223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:48.411789 3223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:48.417316 3232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:48.417488 3228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:48.417373 3223 server_base.cc:1034] running on GCE node
W20260808 02:17:48.417300 3230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:48.417811 3223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:48.418068 3223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:48.419240 3223 hybrid_clock.cc:648] HybridClock initialized: now 1786155468419210 us; error 50 us; skew 500 ppm
Aug 08 02:17:48 dist-test-slave-3h5h krb5kdc[2671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155468, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:48.422263 3223 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:48.422690 3223 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:48.422767 3223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:48.422914 3223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:48.423991 3223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "1b919f564f51ee179b9ca41c5c4cf758"
server_key_iv: "84365433e4a62c5b8c35c968c70be225"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.424353 3223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "1b919f564f51ee179b9ca41c5c4cf758"
server_key_iv: "84365433e4a62c5b8c35c968c70be225"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.425940 3223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:48.426872 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.427166 3223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260808 02:17:48.427277 3223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96"
format_stamp: "Formatted at 2026-08-08 02:17:48 on dist-test-slave-3h5h"
server_key: "1b919f564f51ee179b9ca41c5c4cf758"
server_key_iv: "84365433e4a62c5b8c35c968c70be225"
server_key_version: "encryptionkey@0"
I20260808 02:17:48.427356 3223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:48.437623 3223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:48.440062 3223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:48.440243 3223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:48.440497 3223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:48.440927 3223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:48.441045 3223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.441083 3223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:48.441115 3223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:48.447204 3223 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:40495
I20260808 02:17:48.447252 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:40495 every 8 connection(s)
I20260808 02:17:48.447570 3223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:17:48 dist-test-slave-3h5h krb5kdc[2671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155468, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:48.456804 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 3223
I20260808 02:17:48.456939 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:48.457144 29382 external_mini_cluster.cc:1428] Setting key 31bbb57c657bc43db1b68e367666dd72
I20260808 02:17:48.458334 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.448421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:32847 (local address 127.28.177.190:38103)
0808 02:17:48.448474 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.448476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.448671 (+ 195us) server_negotiation.cc:407] Connection header received
0808 02:17:48.449251 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.449252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.449262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.449325 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:48.450325 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.450825 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.451890 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.452110 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.452173 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.454644 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:48.454645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:48.454645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:48.454654 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:48.456479 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.457188 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.457189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.457207 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.457287 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.457591 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.457592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.457592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.457719 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:48.457800 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.458155 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.458245 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:17:48.458523 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.448367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38103 (local address 127.28.177.131:32847)
0808 02:17:48.448544 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.448588 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.449148 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.449399 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.449402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.449725 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:48.450213 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.450218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.450926 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.450927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.451792 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.451795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.452215 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.452216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.452324 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.452825 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:48.452889 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:48.454506 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:48.456639 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:48.456648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:48.456650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:48.457102 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:48.457370 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:48.457371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:48.457371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:48.457498 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:48.457868 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:48.457874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:48.458005 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.458115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.458282 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260808 02:17:48.458995 3397 heartbeater.cc:347] Connected to a master server at 127.28.177.190:38103
I20260808 02:17:48.459131 3397 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:48.459297 3397 heartbeater.cc:511] Master 127.28.177.190:38103 requested a full tablet report, sending...
I20260808 02:17:48.459714 2738 ts_manager.cc:194] Registered new tserver with Master: 32c28128fdb14d7ea1aa3db1ad0e2c96 (127.28.177.131:40495)
I20260808 02:17:48.460140 2738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:32847
I20260808 02:17:48.469425 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:17:48.478075 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.471009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.28.177.190:38103)
0808 02:17:48.471093 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.471096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.471221 (+ 125us) server_negotiation.cc:407] Connection header received
0808 02:17:48.471268 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.471269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.471280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.471339 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:48.472032 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.472655 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.473538 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.473767 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.473835 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.475565 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:48.475567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:48.475567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:48.475581 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:48.477018 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.477364 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.477365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.477365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.477417 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:48.477599 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:48.477600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:48.477600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:48.477744 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:48.477818 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.477947 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.478006 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:17:48.479897 2738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:48.481802 2738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:48.489189 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.484644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:39257 (local address 127.0.0.1:37514)
0808 02:17:48.485033 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.485076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.485151 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.485545 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.485546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.485548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:48.485609 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.485611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.487100 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.487101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.488094 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.488096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.488898 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.488899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.488967 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.488987 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.489058 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.489110 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":94,"threads_started":1}
I20260808 02:17:48.489269 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.484705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.28.177.131:40495)
0808 02:17:48.485008 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.485014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.485246 (+ 232us) server_negotiation.cc:407] Connection header received
0808 02:17:48.485286 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.485288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.485302 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.485408 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:48.485742 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.486554 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.487947 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.488750 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.488846 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.488937 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.489133 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.489177 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":93,"threads_started":1}
I20260808 02:17:48.489792 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.484980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.28.177.129:45921)
0808 02:17:48.485125 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.485129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.485622 (+ 493us) server_negotiation.cc:407] Connection header received
0808 02:17:48.485671 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.485673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.485689 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.485766 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:48.485996 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.487163 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.488615 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.489383 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.489457 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.489515 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.489668 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.489715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260808 02:17:48.489825 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.484766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.28.177.130:39257)
0808 02:17:48.485326 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.485332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.485342 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:48.485386 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.485389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.485407 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.485544 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:48.485673 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.486969 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.488176 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.488758 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.488832 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.489678 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.489703 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.489743 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":162,"threads_started":1}
I20260808 02:17:48.489311 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.484613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:40495 (local address 127.0.0.1:51166)
0808 02:17:48.485028 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.485151 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.485218 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.485421 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.485425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.485428 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:48.485531 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.485534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.486656 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.486658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.487851 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.487854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.488866 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.488867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.488970 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.488986 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.489110 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.489163 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":75,"threads_started":1}
I20260808 02:17:48.489703 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.484891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45921 (local address 127.0.0.1:54780)
0808 02:17:48.485301 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.485310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.485367 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.485866 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.485866 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.485868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:48.485903 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.485904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.487272 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.487273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.488507 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.488510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.489474 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.489475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.489547 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.489562 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.489625 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.489656 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":63,"threads_started":1}
I20260808 02:17:48.490767 3311 tablet_service.cc:1467] Processing CreateTablet for tablet 02dc45eda3024894bafc4fdc6bb8aa68 (DEFAULT_TABLE table=test-table [id=6b744790eff84cf0ac9742aca5df1d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:48.490767 2917 tablet_service.cc:1467] Processing CreateTablet for tablet 02dc45eda3024894bafc4fdc6bb8aa68 (DEFAULT_TABLE table=test-table [id=6b744790eff84cf0ac9742aca5df1d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:48.490896 3125 tablet_service.cc:1467] Processing CreateTablet for tablet 02dc45eda3024894bafc4fdc6bb8aa68 (DEFAULT_TABLE table=test-table [id=6b744790eff84cf0ac9742aca5df1d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:48.491147 2917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dc45eda3024894bafc4fdc6bb8aa68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:48.491147 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dc45eda3024894bafc4fdc6bb8aa68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:48.491336 3125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dc45eda3024894bafc4fdc6bb8aa68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:48.494405 3428 tablet_bootstrap.cc:492] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Bootstrap starting.
I20260808 02:17:48.494760 3429 tablet_bootstrap.cc:492] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: Bootstrap starting.
I20260808 02:17:48.495091 3430 tablet_bootstrap.cc:492] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Bootstrap starting.
I20260808 02:17:48.495302 3428 tablet_bootstrap.cc:654] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:48.495491 3429 tablet_bootstrap.cc:654] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:48.495770 3430 tablet_bootstrap.cc:654] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:48.496069 3428 log.cc:824] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:48.496215 3429 log.cc:824] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:48.496519 3430 log.cc:824] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:48.497067 3429 tablet_bootstrap.cc:492] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: No bootstrap required, opened a new log
I20260808 02:17:48.497072 3428 tablet_bootstrap.cc:492] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: No bootstrap required, opened a new log
I20260808 02:17:48.497159 3429 ts_tablet_manager.cc:1397] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:48.497159 3428 ts_tablet_manager.cc:1397] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:17:48.497468 3430 tablet_bootstrap.cc:492] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: No bootstrap required, opened a new log
I20260808 02:17:48.497546 3430 ts_tablet_manager.cc:1397] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:17:48.498935 3429 raft_consensus.cc:348] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.499105 3429 raft_consensus.cc:374] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:48.499171 3429 raft_consensus.cc:729] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903a8b29ff8c40a596bd569894d4c68e, State: Initialized, Role: FOLLOWER
I20260808 02:17:48.499349 3429 consensus_queue.cc:260] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.499466 3428 raft_consensus.cc:348] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.499682 3429 ts_tablet_manager.cc:1428] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 02:17:48.499682 3020 heartbeater.cc:503] Master 127.28.177.190:38103 was elected leader, sending a full tablet report...
I20260808 02:17:48.499769 3428 raft_consensus.cc:374] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:48.499832 3428 raft_consensus.cc:729] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b06ff702e8a406b8183840f246e97c2, State: Initialized, Role: FOLLOWER
I20260808 02:17:48.499852 3430 raft_consensus.cc:348] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.500006 3430 raft_consensus.cc:374] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:48.499985 3428 consensus_queue.cc:260] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.500041 3430 raft_consensus.cc:729] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c28128fdb14d7ea1aa3db1ad0e2c96, State: Initialized, Role: FOLLOWER
I20260808 02:17:48.500180 3430 consensus_queue.cc:260] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.500363 3428 ts_tablet_manager.cc:1428] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:48.500373 3218 heartbeater.cc:503] Master 127.28.177.190:38103 was elected leader, sending a full tablet report...
I20260808 02:17:48.500571 3430 ts_tablet_manager.cc:1428] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:48.500689 3397 heartbeater.cc:503] Master 127.28.177.190:38103 was elected leader, sending a full tablet report...
W20260808 02:17:48.530965 3219 tablet.cc:2369] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:48.602280 3021 tablet.cc:2369] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:48.698256 3398 tablet.cc:2369] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:48.788259 3436 raft_consensus.cc:482] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:48.788507 3436 raft_consensus.cc:504] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.788962 3436 leader_election.cc:288] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b06ff702e8a406b8183840f246e97c2 (127.28.177.130:39257), 903a8b29ff8c40a596bd569894d4c68e (127.28.177.129:45921)
I20260808 02:17:48.793233 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.789565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:55247 (local address 127.28.177.129:45921)
0808 02:17:48.789649 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.789651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.789737 (+ 86us) server_negotiation.cc:407] Connection header received
0808 02:17:48.789802 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.789804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.789814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.789877 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:48.790103 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.790780 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.792211 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.792792 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.792918 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.792927 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.793107 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.793157 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:17:48.793215 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.789440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45921 (local address 127.28.177.131:55247)
0808 02:17:48.789656 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.789666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.789728 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.789906 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.789906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.789908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:48.789959 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.789960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.790857 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.790858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.792070 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.792073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.792922 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.792923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.793012 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.793017 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.793079 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.793122 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":36,"thread_start_us":156,"threads_started":1}
I20260808 02:17:48.793272 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.789388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:35207 (local address 127.28.177.130:39257)
0808 02:17:48.789460 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.789462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.789476 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:17:48.789547 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.789548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.789558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.789655 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:48.789833 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.790685 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.792243 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.792807 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.792917 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.792943 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.793165 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.793206 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:17:48.793323 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.789274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:39257 (local address 127.28.177.131:35207)
0808 02:17:48.789330 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.789344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:48.789398 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:48.789663 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:48.789665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:48.789667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:48.789750 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.789752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.790770 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.790771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.792141 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:48.792143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.792922 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.792923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:48.793016 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.793018 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:17:48.793154 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 02:17:48.793190 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":21}
I20260808 02:17:48.793748 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dc45eda3024894bafc4fdc6bb8aa68" candidate_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b06ff702e8a406b8183840f246e97c2" is_pre_election: true
I20260808 02:17:48.793747 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dc45eda3024894bafc4fdc6bb8aa68" candidate_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903a8b29ff8c40a596bd569894d4c68e" is_pre_election: true
I20260808 02:17:48.793915 3152 raft_consensus.cc:2454] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c28128fdb14d7ea1aa3db1ad0e2c96 in term 0.
I20260808 02:17:48.793913 2948 raft_consensus.cc:2454] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c28128fdb14d7ea1aa3db1ad0e2c96 in term 0.
I20260808 02:17:48.794157 3248 leader_election.cc:302] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c28128fdb14d7ea1aa3db1ad0e2c96, 9b06ff702e8a406b8183840f246e97c2; no voters:
I20260808 02:17:48.794297 3436 raft_consensus.cc:2781] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:48.794354 3436 raft_consensus.cc:482] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:48.794384 3436 raft_consensus.cc:3037] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:48.795092 3436 raft_consensus.cc:504] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.795228 3436 leader_election.cc:288] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [CANDIDATE]: Term 1 election: Requested vote from peers 9b06ff702e8a406b8183840f246e97c2 (127.28.177.130:39257), 903a8b29ff8c40a596bd569894d4c68e (127.28.177.129:45921)
I20260808 02:17:48.795413 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dc45eda3024894bafc4fdc6bb8aa68" candidate_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903a8b29ff8c40a596bd569894d4c68e"
I20260808 02:17:48.795425 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dc45eda3024894bafc4fdc6bb8aa68" candidate_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b06ff702e8a406b8183840f246e97c2"
I20260808 02:17:48.795508 2948 raft_consensus.cc:3037] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:48.795504 3152 raft_consensus.cc:3037] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:48.796219 3152 raft_consensus.cc:2454] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c28128fdb14d7ea1aa3db1ad0e2c96 in term 1.
I20260808 02:17:48.796288 2948 raft_consensus.cc:2454] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c28128fdb14d7ea1aa3db1ad0e2c96 in term 1.
I20260808 02:17:48.796423 3248 leader_election.cc:302] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c28128fdb14d7ea1aa3db1ad0e2c96, 9b06ff702e8a406b8183840f246e97c2; no voters:
I20260808 02:17:48.796573 3436 raft_consensus.cc:2781] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:48.796777 3436 raft_consensus.cc:686] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 1 LEADER]: Becoming Leader. State: Replica: 32c28128fdb14d7ea1aa3db1ad0e2c96, State: Running, Role: LEADER
I20260808 02:17:48.796917 3436 consensus_queue.cc:237] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } }
I20260808 02:17:48.797946 2737 catalog_manager.cc:5495] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32c28128fdb14d7ea1aa3db1ad0e2c96 (127.28.177.131). New cstate: current_term: 1 leader_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c28128fdb14d7ea1aa3db1ad0e2c96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:48.846657 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.843809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.28.177.131:40495)
0808 02:17:48.843899 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:48.843901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:48.843910 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:48.843951 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:48.843952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:48.843972 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:48.844071 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:48.844259 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.844743 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.845696 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:48.845907 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:48.845978 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:48.846197 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:48.846312 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:48.846386 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:48.846480 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:17:48.846501 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:17:48.849097 3152 raft_consensus.cc:1261] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 1 FOLLOWER]: Refusing update from remote peer 32c28128fdb14d7ea1aa3db1ad0e2c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:48.849185 2948 raft_consensus.cc:1261] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 1 FOLLOWER]: Refusing update from remote peer 32c28128fdb14d7ea1aa3db1ad0e2c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:48.849628 3436 consensus_queue.cc:1035] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903a8b29ff8c40a596bd569894d4c68e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:48.849892 3438 consensus_queue.cc:1035] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b06ff702e8a406b8183840f246e97c2" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 39257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:48.852569 3446 mvcc.cc:205] Tried to move back new op lower bound from 7316092800401838080 to 7316092800193064960. Current Snapshot: MvccSnapshot[applied={T|T < 7316092800401838080}]
I20260808 02:17:48.853160 3447 mvcc.cc:205] Tried to move back new op lower bound from 7316092800401838080 to 7316092800193064960. Current Snapshot: MvccSnapshot[applied={T|T < 7316092800401838080}]
I20260808 02:17:48.855611 2737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:48.855711 2737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:48.856654 2737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 496a0e9a29c24a1d8b6b052cc7df154c: Sending DeleteTablet for 3 replicas of tablet 02dc45eda3024894bafc4fdc6bb8aa68
I20260808 02:17:48.857059 3125 tablet_service.cc:1514] Processing DeleteTablet for tablet 02dc45eda3024894bafc4fdc6bb8aa68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:48 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:37514
I20260808 02:17:48.857117 3311 tablet_service.cc:1514] Processing DeleteTablet for tablet 02dc45eda3024894bafc4fdc6bb8aa68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:48 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:51166
I20260808 02:17:48.857262 2917 tablet_service.cc:1514] Processing DeleteTablet for tablet 02dc45eda3024894bafc4fdc6bb8aa68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:48 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:54780
I20260808 02:17:48.857378 3458 tablet_replica.cc:330] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: stopping tablet replica
I20260808 02:17:48.857504 3458 raft_consensus.cc:2229] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:48.857543 3461 tablet_replica.cc:330] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e: stopping tablet replica
I20260808 02:17:48.857648 3461 raft_consensus.cc:2229] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:48.857704 3458 raft_consensus.cc:2258] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:48.857843 3461 raft_consensus.cc:2258] T 02dc45eda3024894bafc4fdc6bb8aa68 P 903a8b29ff8c40a596bd569894d4c68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:48.857977 3459 tablet_replica.cc:330] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: stopping tablet replica
I20260808 02:17:48.858031 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 2833
I20260808 02:17:48.858289 3459 raft_consensus.cc:2229] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:17:48.858422 3459 raft_consensus.cc:2258] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:48.858455 3458 ts_tablet_manager.cc:1905] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:48.860555 3458 ts_tablet_manager.cc:1918] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:48.860613 3459 ts_tablet_manager.cc:1905] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:48.860638 3458 log.cc:1196] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/02dc45eda3024894bafc4fdc6bb8aa68
I20260808 02:17:48.860970 3458 ts_tablet_manager.cc:1939] T 02dc45eda3024894bafc4fdc6bb8aa68 P 9b06ff702e8a406b8183840f246e97c2: Deleting consensus metadata
I20260808 02:17:48.861575 2721 catalog_manager.cc:4841] TS 9b06ff702e8a406b8183840f246e97c2 (127.28.177.130:39257): tablet 02dc45eda3024894bafc4fdc6bb8aa68 (table test-table [id=6b744790eff84cf0ac9742aca5df1d36]) successfully deleted
I20260808 02:17:48.862757 3459 ts_tablet_manager.cc:1918] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:48.862824 3459 log.cc:1196] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/02dc45eda3024894bafc4fdc6bb8aa68
I20260808 02:17:48.863104 3459 ts_tablet_manager.cc:1939] T 02dc45eda3024894bafc4fdc6bb8aa68 P 32c28128fdb14d7ea1aa3db1ad0e2c96: Deleting consensus metadata
I20260808 02:17:48.863607 2719 catalog_manager.cc:4841] TS 32c28128fdb14d7ea1aa3db1ad0e2c96 (127.28.177.131:40495): tablet 02dc45eda3024894bafc4fdc6bb8aa68 (table test-table [id=6b744790eff84cf0ac9742aca5df1d36]) successfully deleted
W20260808 02:17:48.871177 2721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:45921 (error 108)
I20260808 02:17:48.871368 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 3025
I20260808 02:17:48.878232 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:48.871761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45921 (local address 127.0.0.1:53288)
0808 02:17:48.878095 (+ 6334us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:48.878170 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6300}
W20260808 02:17:48.882824 2721 catalog_manager.cc:4584] TS 903a8b29ff8c40a596bd569894d4c68e (127.28.177.129:45921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02dc45eda3024894bafc4fdc6bb8aa68: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45921: connect: Connection refused (error 111)
I20260808 02:17:48.888942 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 3223
I20260808 02:17:48.901546 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 2693
2026-08-08T02:17:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2540 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T02:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:48Z Disabled control of system clock
I20260808 02:17:48.939050 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:46437
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:17:49.024605 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:49.024865 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:49.024890 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:49.026063 3466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:49.026110 3466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:49.026124 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:49.026136 3466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:49.026149 3466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:49.027513 3466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:46437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:49.027736 3466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:49.027992 3466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:49.033607 3474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.033705 3466 server_base.cc:1034] running on GCE node
W20260808 02:17:49.033561 3473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:49.033561 3476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.034130 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:49.034482 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:49.035646 3466 hybrid_clock.cc:648] HybridClock initialized: now 1786155469035636 us; error 32 us; skew 500 ppm
I20260808 02:17:49.037040 3466 webserver.cc:460] Webserver started at http://127.28.177.190:41739/ using document root <none> and password file <none>
I20260808 02:17:49.037339 3466 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:49.037413 3466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:49.037609 3466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:49.038720 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "593303c37aef4260a4d6ad531bdaafbb"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "20b2211bcfe5964f9c2654262239dd80"
server_key_iv: "2b74f00aa45a2cd1737a92593b0aaed3"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.039103 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "593303c37aef4260a4d6ad531bdaafbb"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "20b2211bcfe5964f9c2654262239dd80"
server_key_iv: "2b74f00aa45a2cd1737a92593b0aaed3"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.040463 3466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:49.041512 3485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.041754 3466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:49.041869 3466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "593303c37aef4260a4d6ad531bdaafbb"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "20b2211bcfe5964f9c2654262239dd80"
server_key_iv: "2b74f00aa45a2cd1737a92593b0aaed3"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.041949 3466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:49.045492 3466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:49.045773 3466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:49.045919 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:49.050278 3466 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:46437
I20260808 02:17:49.050340 3559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:46437 every 8 connection(s)
I20260808 02:17:49.050691 3466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:49.051440 3562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:49.054159 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 3466
I20260808 02:17:49.054275 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:49.054472 29382 external_mini_cluster.cc:1428] Setting key 0a980b31e5cfbc65b60c7e0c0813f7aa
I20260808 02:17:49.055236 3562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb: Bootstrap starting.
I20260808 02:17:49.055961 3562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:49.056348 3562 log.cc:824] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:49.057262 3562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb: No bootstrap required, opened a new log
I20260808 02:17:49.057497 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.055075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.28.177.190:46437)
0808 02:17:49.055297 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.055324 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.055361 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:17:49.055802 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.055808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.055848 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.055974 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.056807 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.056813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.056814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.056820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.056909 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.056922 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.056990 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.057097 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.057286 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
I20260808 02:17:49.059211 3562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593303c37aef4260a4d6ad531bdaafbb" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46437 } }
I20260808 02:17:49.059382 3562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:49.059430 3562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593303c37aef4260a4d6ad531bdaafbb, State: Initialized, Role: FOLLOWER
I20260808 02:17:49.059561 3562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593303c37aef4260a4d6ad531bdaafbb" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46437 } }
I20260808 02:17:49.059655 3562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:49.059700 3562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:49.059751 3562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:49.060314 3562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593303c37aef4260a4d6ad531bdaafbb" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46437 } }
I20260808 02:17:49.060432 3562 leader_election.cc:302] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 593303c37aef4260a4d6ad531bdaafbb; no voters:
I20260808 02:17:49.060622 3562 leader_election.cc:288] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:49.060710 3567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:49.060916 3567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [term 1 LEADER]: Becoming Leader. State: Replica: 593303c37aef4260a4d6ad531bdaafbb, State: Running, Role: LEADER
I20260808 02:17:49.060977 3562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:49.061062 3567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593303c37aef4260a4d6ad531bdaafbb" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46437 } }
I20260808 02:17:49.061501 3568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "593303c37aef4260a4d6ad531bdaafbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593303c37aef4260a4d6ad531bdaafbb" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46437 } } }
I20260808 02:17:49.061539 3570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 593303c37aef4260a4d6ad531bdaafbb. Latest consensus state: current_term: 1 leader_uuid: "593303c37aef4260a4d6ad531bdaafbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593303c37aef4260a4d6ad531bdaafbb" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46437 } } }
I20260808 02:17:49.061718 3570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:49.061965 3568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:49.062177 3581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:49.062739 3581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:49.064534 3581 catalog_manager.cc:1347] Generated new cluster ID: e246b38cbdbf4285ad7d19bc280e163a
I20260808 02:17:49.064591 3581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:49.079885 3581 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:49.080106 3581 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:49.087809 3581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb: Generated new TSK 0
I20260808 02:17:49.088056 3581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:17:49.095019 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:46437
--builtin_ntp_servers=127.28.177.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:17:49.199462 3595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:49.199682 3595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:49.199705 3595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:49.200977 3595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:49.201030 3595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:49.201057 3595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:49.202507 3595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:46437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:49.202776 3595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:49.203132 3595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:49.209467 3601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:49.209649 3608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:49.209458 3600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.209925 3595 server_base.cc:1034] running on GCE node
I20260808 02:17:49.210223 3595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:49.210561 3595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:49.211745 3595 hybrid_clock.cc:648] HybridClock initialized: now 1786155469211732 us; error 28 us; skew 500 ppm
I20260808 02:17:49.213331 3595 webserver.cc:460] Webserver started at http://127.28.177.129:36955/ using document root <none> and password file <none>
I20260808 02:17:49.213656 3595 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:49.213719 3595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:49.213862 3595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:49.214977 3595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "4ed4fd9ac54d403f814dd0af5657c2c9"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "10d630aae9c713a26d63b9c081e9d429"
server_key_iv: "59524530f2738a4cea612d008421ded2"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.215359 3595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "4ed4fd9ac54d403f814dd0af5657c2c9"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "10d630aae9c713a26d63b9c081e9d429"
server_key_iv: "59524530f2738a4cea612d008421ded2"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.216778 3595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:49.217655 3614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.217818 3595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:49.217909 3595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "4ed4fd9ac54d403f814dd0af5657c2c9"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "10d630aae9c713a26d63b9c081e9d429"
server_key_iv: "59524530f2738a4cea612d008421ded2"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.217990 3595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:49.221544 3595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:49.221841 3595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:49.221990 3595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:49.222245 3595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:49.222648 3595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:49.222687 3595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.222707 3595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:49.222719 3595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.229028 3595 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:43167
I20260808 02:17:49.229099 3780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:43167 every 8 connection(s)
I20260808 02:17:49.229425 3595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:49.232431 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 3595
I20260808 02:17:49.232537 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:49.232725 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.230243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:59033 (local address 127.28.177.190:46437)
0808 02:17:49.230294 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.230296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.230530 (+ 234us) server_negotiation.cc:407] Connection header received
0808 02:17:49.231000 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.231002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.231008 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.231080 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.232336 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.232338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.232338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.232342 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.232358 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.232362 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.232421 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.232602 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.232622 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:17:49.232739 29382 external_mini_cluster.cc:1428] Setting key 3afc1a80c3ed3988474993eaabc3fe03
I20260808 02:17:49.232892 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.230208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:46437 (local address 127.28.177.129:59033)
0808 02:17:49.230390 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.230431 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.230899 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.231134 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.231138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.231146 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.232115 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.232116 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.232156 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.232217 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.232443 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.232450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.232453 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.232557 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.232631 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260808 02:17:49.233546 3781 heartbeater.cc:347] Connected to a master server at 127.28.177.190:46437
I20260808 02:17:49.233695 3781 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:49.233873 3781 heartbeater.cc:511] Master 127.28.177.190:46437 requested a full tablet report, sending...
I20260808 02:17:49.233925 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:46437
--builtin_ntp_servers=127.28.177.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:17:49.234597 3508 ts_manager.cc:194] Registered new tserver with Master: 4ed4fd9ac54d403f814dd0af5657c2c9 (127.28.177.129:43167)
W20260808 02:17:49.320458 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:49.320708 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:49.320748 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:49.322078 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:49.322146 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:49.322191 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:49.323837 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:46437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:49.324110 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:49.324419 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:49.329933 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.330184 3785 server_base.cc:1034] running on GCE node
W20260808 02:17:49.329939 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:49.329916 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.330543 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:49.330789 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:49.331974 3785 hybrid_clock.cc:648] HybridClock initialized: now 1786155469331965 us; error 51 us; skew 500 ppm
I20260808 02:17:49.333575 3785 webserver.cc:460] Webserver started at http://127.28.177.130:42473/ using document root <none> and password file <none>
I20260808 02:17:49.333863 3785 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:49.333941 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:49.334125 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:49.335448 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "79e074e5ab2346a5a005baf24b353a81"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "d169b982597ffb4bce7b3d43727a36ab"
server_key_iv: "ceecf85f15cf8e6a907e1c6b57441e47"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.335858 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "79e074e5ab2346a5a005baf24b353a81"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "d169b982597ffb4bce7b3d43727a36ab"
server_key_iv: "ceecf85f15cf8e6a907e1c6b57441e47"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.337399 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260808 02:17:49.338300 3803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.338482 3785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:49.338613 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "79e074e5ab2346a5a005baf24b353a81"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "d169b982597ffb4bce7b3d43727a36ab"
server_key_iv: "ceecf85f15cf8e6a907e1c6b57441e47"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.338699 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:49.342707 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:49.343021 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:49.343183 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:49.343436 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:49.343775 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:49.343837 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.343883 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:49.343920 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.351054 3785 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:35847
I20260808 02:17:49.351115 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:35847 every 8 connection(s)
I20260808 02:17:49.351508 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:17:49.355302 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.352458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37553 (local address 127.28.177.190:46437)
0808 02:17:49.352556 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.352559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.352784 (+ 225us) server_negotiation.cc:407] Connection header received
0808 02:17:49.353392 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.353394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.353400 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.353463 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.354902 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.354904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.354904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.354909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.354925 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.354929 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.354989 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.355227 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.355242 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:17:49.355543 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.352431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:46437 (local address 127.28.177.130:37553)
0808 02:17:49.352652 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.352689 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.353271 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.353627 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.353631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.353641 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.354699 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.354699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.354737 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.354767 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.355116 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.355124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.355127 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.355217 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.355276 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260808 02:17:49.356247 3964 heartbeater.cc:347] Connected to a master server at 127.28.177.190:46437
I20260808 02:17:49.356485 3964 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:49.356691 3964 heartbeater.cc:511] Master 127.28.177.190:46437 requested a full tablet report, sending...
I20260808 02:17:49.357210 3508 ts_manager.cc:194] Registered new tserver with Master: 79e074e5ab2346a5a005baf24b353a81 (127.28.177.130:35847)
I20260808 02:17:49.360076 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 3785
I20260808 02:17:49.360210 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:49.360491 29382 external_mini_cluster.cc:1428] Setting key fb4393a87355d161e451176958501c81
I20260808 02:17:49.361589 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:46437
--builtin_ntp_servers=127.28.177.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:17:49.446314 3969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:49.446527 3969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:49.446549 3969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:49.447724 3969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:49.447768 3969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:49.447794 3969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:49.449278 3969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:46437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:49.449550 3969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:49.449905 3969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:49.455468 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:49.455488 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.455528 3969 server_base.cc:1034] running on GCE node
W20260808 02:17:49.455482 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:49.456146 3969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:49.456425 3969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:49.457590 3969 hybrid_clock.cc:648] HybridClock initialized: now 1786155469457566 us; error 37 us; skew 500 ppm
I20260808 02:17:49.458940 3969 webserver.cc:460] Webserver started at http://127.28.177.131:46471/ using document root <none> and password file <none>
I20260808 02:17:49.459255 3969 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:49.459308 3969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:49.459411 3969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:49.460549 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "7928cf797acd409e856c0cca9cd44b20"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "f42b08a5f619827e4ec184f087d0e6e9"
server_key_iv: "c2363bc57185e885a3279fc1c062eb3b"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.461067 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "7928cf797acd409e856c0cca9cd44b20"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "f42b08a5f619827e4ec184f087d0e6e9"
server_key_iv: "c2363bc57185e885a3279fc1c062eb3b"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.462706 3969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:17:49.463632 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.463835 3969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:49.463958 3969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "7928cf797acd409e856c0cca9cd44b20"
format_stamp: "Formatted at 2026-08-08 02:17:49 on dist-test-slave-3h5h"
server_key: "f42b08a5f619827e4ec184f087d0e6e9"
server_key_iv: "c2363bc57185e885a3279fc1c062eb3b"
server_key_version: "encryptionkey@0"
I20260808 02:17:49.464128 3969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:49.468039 3969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:49.468325 3969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:49.468477 3969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:49.468699 3969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:49.469134 3969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:49.469195 3969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.469240 3969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:49.469270 3969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:49.475850 3969 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:40533
I20260808 02:17:49.475893 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:40533 every 8 connection(s)
I20260808 02:17:49.476292 3969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:17:49.477437 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 3969
I20260808 02:17:49.477558 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:49.477754 29382 external_mini_cluster.cc:1428] Setting key de01228fdc33a85464ebaedaadfaccc3
I20260808 02:17:49.480051 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.477260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:52135 (local address 127.28.177.190:46437)
0808 02:17:49.477336 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.477338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.477560 (+ 222us) server_negotiation.cc:407] Connection header received
0808 02:17:49.478269 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.478270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.478276 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.478340 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.479716 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.479718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.479718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.479722 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.479737 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.479741 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.479790 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.479907 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.479919 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:17:49.480255 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.477130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:46437 (local address 127.28.177.131:52135)
0808 02:17:49.477378 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.477422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.478115 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.478437 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.478442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.478454 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.479567 (+ 1113us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.479567 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.479599 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.479629 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.479807 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.479812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.479815 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.479921 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.480024 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260808 02:17:49.480749 4150 heartbeater.cc:347] Connected to a master server at 127.28.177.190:46437
I20260808 02:17:49.480881 4150 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:49.481104 4150 heartbeater.cc:511] Master 127.28.177.190:46437 requested a full tablet report, sending...
I20260808 02:17:49.481506 3508 ts_manager.cc:194] Registered new tserver with Master: 7928cf797acd409e856c0cca9cd44b20 (127.28.177.131:40533)
I20260808 02:17:49.489895 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:17:49.492653 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.491429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.28.177.190:46437)
0808 02:17:49.491484 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.491486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.491616 (+ 130us) server_negotiation.cc:407] Connection header received
0808 02:17:49.491677 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.491678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.491683 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.491731 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.492317 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.492318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.492318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.492322 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.492338 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.492342 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.492391 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.492529 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.492541 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:17:49.494589 3508 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:49.496249 3508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:49.500229 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.498683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:35847 (local address 127.0.0.1:49940)
0808 02:17:49.498813 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.498858 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.498918 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.499251 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.499258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.499265 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.499796 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.499796 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.499813 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.499837 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.500045 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.500052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.500056 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.500129 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.500145 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260808 02:17:49.500541 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.498897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.28.177.130:35847)
0808 02:17:49.499081 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.499087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.499098 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:49.499129 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.499131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.499144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.499249 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.499902 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.499907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.499907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.499912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.499943 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.499959 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.500051 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.500139 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.500155 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260808 02:17:49.501336 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.498974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.28.177.129:43167)
0808 02:17:49.499303 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.499306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.499315 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:49.499339 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.499341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.499348 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.499452 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.500979 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.500991 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.500991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.501010 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.501048 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.501082 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.501162 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.501231 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.501248 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":206,"threads_started":1}
I20260808 02:17:49.501322 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.498694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:43167 (local address 127.0.0.1:49058)
0808 02:17:49.499113 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.499125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.499178 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.500020 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.500021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.500024 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.500820 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.500821 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.500859 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.500874 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.501156 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.501157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.501157 (+ 0us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.501222 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.501238 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":106,"threads_started":1}
I20260808 02:17:49.501334 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.498701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:40533 (local address 127.0.0.1:59564)
0808 02:17:49.499826 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.499836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.499916 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.500489 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.500490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.500493 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.500925 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.500926 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.500931 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.500942 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.501178 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.501179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.501180 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.501225 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.501240 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":112,"threads_started":1}
I20260808 02:17:49.501461 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.498974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.28.177.131:40533)
0808 02:17:49.499200 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.499205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.499917 (+ 712us) server_negotiation.cc:407] Connection header received
0808 02:17:49.499985 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.499989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.500000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.500158 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.501012 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.501021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.501021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.501026 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.501061 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.501084 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.501150 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.501368 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.501385 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260808 02:17:49.501317 3872 tablet_service.cc:1467] Processing CreateTablet for tablet 6a679ec659aa40748fda5541963c017b (DEFAULT_TABLE table=test-table [id=047c70bd0ec5435d8da3f4b31b7b2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:49.501729 3872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a679ec659aa40748fda5541963c017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:49.502461 3685 tablet_service.cc:1467] Processing CreateTablet for tablet 6a679ec659aa40748fda5541963c017b (DEFAULT_TABLE table=test-table [id=047c70bd0ec5435d8da3f4b31b7b2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:49.502811 3685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a679ec659aa40748fda5541963c017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:49.504597 4165 tablet_bootstrap.cc:492] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Bootstrap starting.
I20260808 02:17:49.505376 4165 tablet_bootstrap.cc:654] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:49.505676 4165 log.cc:824] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:49.505658 4061 tablet_service.cc:1467] Processing CreateTablet for tablet 6a679ec659aa40748fda5541963c017b (DEFAULT_TABLE table=test-table [id=047c70bd0ec5435d8da3f4b31b7b2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:49.506054 4061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a679ec659aa40748fda5541963c017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:49.506139 4169 tablet_bootstrap.cc:492] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: Bootstrap starting.
I20260808 02:17:49.506735 4169 tablet_bootstrap.cc:654] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:49.506983 4165 tablet_bootstrap.cc:492] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: No bootstrap required, opened a new log
I20260808 02:17:49.507093 4165 ts_tablet_manager.cc:1397] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:17:49.507092 4169 log.cc:824] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:49.508010 4169 tablet_bootstrap.cc:492] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: No bootstrap required, opened a new log
I20260808 02:17:49.508091 4169 ts_tablet_manager.cc:1397] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:17:49.509023 4171 tablet_bootstrap.cc:492] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Bootstrap starting.
I20260808 02:17:49.509558 4171 tablet_bootstrap.cc:654] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:49.509881 4171 log.cc:824] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:49.510049 4165 raft_consensus.cc:348] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.510228 4165 raft_consensus.cc:374] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:49.510273 4165 raft_consensus.cc:729] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e074e5ab2346a5a005baf24b353a81, State: Initialized, Role: FOLLOWER
I20260808 02:17:49.510401 4165 consensus_queue.cc:260] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.510735 4165 ts_tablet_manager.cc:1428] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 02:17:49.511013 4171 tablet_bootstrap.cc:492] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: No bootstrap required, opened a new log
I20260808 02:17:49.511097 4171 ts_tablet_manager.cc:1397] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:17:49.511230 4169 raft_consensus.cc:348] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.511353 4169 raft_consensus.cc:374] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:49.511374 4169 raft_consensus.cc:729] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed4fd9ac54d403f814dd0af5657c2c9, State: Initialized, Role: FOLLOWER
I20260808 02:17:49.511523 4169 consensus_queue.cc:260] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.511798 4169 ts_tablet_manager.cc:1428] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 02:17:49.511838 3781 heartbeater.cc:503] Master 127.28.177.190:46437 was elected leader, sending a full tablet report...
I20260808 02:17:49.512990 3964 heartbeater.cc:503] Master 127.28.177.190:46437 was elected leader, sending a full tablet report...
I20260808 02:17:49.513513 4171 raft_consensus.cc:348] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.513654 4171 raft_consensus.cc:374] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:49.513705 4171 raft_consensus.cc:729] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7928cf797acd409e856c0cca9cd44b20, State: Initialized, Role: FOLLOWER
I20260808 02:17:49.513819 4171 consensus_queue.cc:260] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.514079 4171 ts_tablet_manager.cc:1428] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:17:49.514120 4150 heartbeater.cc:503] Master 127.28.177.190:46437 was elected leader, sending a full tablet report...
W20260808 02:17:49.602214 3966 tablet.cc:2369] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:49.726951 4151 tablet.cc:2369] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:49.730206 3783 tablet.cc:2369] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:49.894690 4175 raft_consensus.cc:482] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:49.894850 4175 raft_consensus.cc:504] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.895393 4175 leader_election.cc:288] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e074e5ab2346a5a005baf24b353a81 (127.28.177.130:35847), 7928cf797acd409e856c0cca9cd44b20 (127.28.177.131:40533)
I20260808 02:17:49.897125 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.895746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:45369 (local address 127.28.177.131:40533)
0808 02:17:49.895809 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.895812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.895939 (+ 127us) server_negotiation.cc:407] Connection header received
0808 02:17:49.895982 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.895983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.895992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.896043 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.896806 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.896807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.896807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.896812 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.896873 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.896878 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.896943 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.897058 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.897070 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:17:49.897159 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.895694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:40533 (local address 127.28.177.129:45369)
0808 02:17:49.895818 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.895831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.895927 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.896094 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.896096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.896100 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.896689 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.896689 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.896698 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.896714 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.896960 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.896961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.896962 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.897039 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.897057 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":47,"threads_started":1}
I20260808 02:17:49.897526 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.895752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:49555 (local address 127.28.177.130:35847)
0808 02:17:49.895854 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.895857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.895867 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:49.895969 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.895971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.895978 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.896136 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.897107 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.897109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.897109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.897114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.897132 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.897137 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.897208 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.897447 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.897462 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:17:49.897534 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.895651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:35847 (local address 127.28.177.129:49555)
0808 02:17:49.895796 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.895808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:49.895867 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:49.896192 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:49.896193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:49.896196 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:49.896918 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:17:49.896919 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:49.896927 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:17:49.896940 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:49.897380 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:49.897381 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:49.897381 (+ 0us) client_negotiation.cc:769] Sending connection context
0808 02:17:49.897436 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:17:49.897451 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260808 02:17:49.897778 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a679ec659aa40748fda5541963c017b" candidate_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7928cf797acd409e856c0cca9cd44b20" is_pre_election: true
I20260808 02:17:49.897882 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a679ec659aa40748fda5541963c017b" candidate_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e074e5ab2346a5a005baf24b353a81" is_pre_election: true
I20260808 02:17:49.897959 4087 raft_consensus.cc:2454] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed4fd9ac54d403f814dd0af5657c2c9 in term 0.
I20260808 02:17:49.898026 3902 raft_consensus.cc:2454] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ed4fd9ac54d403f814dd0af5657c2c9 in term 0.
I20260808 02:17:49.898250 3617 leader_election.cc:302] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed4fd9ac54d403f814dd0af5657c2c9, 7928cf797acd409e856c0cca9cd44b20; no voters:
I20260808 02:17:49.898473 4175 raft_consensus.cc:2781] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:49.898535 4175 raft_consensus.cc:482] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:49.898554 4175 raft_consensus.cc:3037] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:49.899362 4175 raft_consensus.cc:504] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.899531 4175 leader_election.cc:288] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [CANDIDATE]: Term 1 election: Requested vote from peers 79e074e5ab2346a5a005baf24b353a81 (127.28.177.130:35847), 7928cf797acd409e856c0cca9cd44b20 (127.28.177.131:40533)
I20260808 02:17:49.899706 3902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a679ec659aa40748fda5541963c017b" candidate_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e074e5ab2346a5a005baf24b353a81"
I20260808 02:17:49.899729 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a679ec659aa40748fda5541963c017b" candidate_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7928cf797acd409e856c0cca9cd44b20"
I20260808 02:17:49.899792 3902 raft_consensus.cc:3037] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:49.899844 4087 raft_consensus.cc:3037] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:49.900583 3902 raft_consensus.cc:2454] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed4fd9ac54d403f814dd0af5657c2c9 in term 1.
I20260808 02:17:49.900604 4087 raft_consensus.cc:2454] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ed4fd9ac54d403f814dd0af5657c2c9 in term 1.
I20260808 02:17:49.900777 3619 leader_election.cc:302] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed4fd9ac54d403f814dd0af5657c2c9, 79e074e5ab2346a5a005baf24b353a81; no voters:
I20260808 02:17:49.900918 4175 raft_consensus.cc:2781] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:49.901101 4175 raft_consensus.cc:686] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed4fd9ac54d403f814dd0af5657c2c9, State: Running, Role: LEADER
I20260808 02:17:49.901211 4175 consensus_queue.cc:237] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } }
I20260808 02:17:49.902161 3508 catalog_manager.cc:5495] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ed4fd9ac54d403f814dd0af5657c2c9 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ed4fd9ac54d403f814dd0af5657c2c9" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:49.947540 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.946495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.28.177.129:43167)
0808 02:17:49.946563 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:49.946565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:49.946573 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:49.946614 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:49.946615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:49.946629 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:49.946694 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:49.947317 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:49.947318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:49.947319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:49.947323 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:49.947340 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:49.947344 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:49.947403 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:49.947462 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 02:17:49.947475 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:17:49.949565 4087 raft_consensus.cc:1261] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 1 FOLLOWER]: Refusing update from remote peer 4ed4fd9ac54d403f814dd0af5657c2c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:49.949571 3902 raft_consensus.cc:1261] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 1 FOLLOWER]: Refusing update from remote peer 4ed4fd9ac54d403f814dd0af5657c2c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:49.949962 4175 consensus_queue.cc:1035] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e074e5ab2346a5a005baf24b353a81" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:49.950197 4181 consensus_queue.cc:1035] T 6a679ec659aa40748fda5541963c017b P 4ed4fd9ac54d403f814dd0af5657c2c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7928cf797acd409e856c0cca9cd44b20" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:49.952656 4186 mvcc.cc:205] Tried to move back new op lower bound from 7316092804910141440 to 7316092804716089344. Current Snapshot: MvccSnapshot[applied={T|T < 7316092804910141440}]
I20260808 02:17:49.952869 4189 mvcc.cc:205] Tried to move back new op lower bound from 7316092804910141440 to 7316092804716089344. Current Snapshot: MvccSnapshot[applied={T|T < 7316092804910141440}]
I20260808 02:17:49.953272 4187 mvcc.cc:205] Tried to move back new op lower bound from 7316092804910141440 to 7316092804716089344. Current Snapshot: MvccSnapshot[applied={T|T < 7316092804910141440}]
I20260808 02:17:49.955569 3508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:49.955680 3508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:49.956452 3508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 593303c37aef4260a4d6ad531bdaafbb: Sending DeleteTablet for 3 replicas of tablet 6a679ec659aa40748fda5541963c017b
I20260808 02:17:49.956794 3872 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a679ec659aa40748fda5541963c017b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:49 UTC) from {username='slave'} at 127.0.0.1:49940
I20260808 02:17:49.956907 3685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a679ec659aa40748fda5541963c017b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:49 UTC) from {username='slave'} at 127.0.0.1:49058
I20260808 02:17:49.956941 4061 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a679ec659aa40748fda5541963c017b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:49 UTC) from {username='slave'} at 127.0.0.1:59564
I20260808 02:17:49.957177 4200 tablet_replica.cc:330] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: stopping tablet replica
I20260808 02:17:49.957330 4200 raft_consensus.cc:2229] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:49.957509 4200 raft_consensus.cc:2258] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:49.957528 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 3595
I20260808 02:17:49.967245 4202 tablet_replica.cc:330] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: stopping tablet replica
I20260808 02:17:49.967228 4200 ts_tablet_manager.cc:1905] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:49.967378 4202 raft_consensus.cc:2229] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:49.967520 4202 raft_consensus.cc:2258] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:49.967988 4202 ts_tablet_manager.cc:1905] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:49.969503 4200 ts_tablet_manager.cc:1918] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:49.969596 4200 log.cc:1196] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/6a679ec659aa40748fda5541963c017b
I20260808 02:17:49.969874 4202 ts_tablet_manager.cc:1918] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:49.969985 4200 ts_tablet_manager.cc:1939] T 6a679ec659aa40748fda5541963c017b P 79e074e5ab2346a5a005baf24b353a81: Deleting consensus metadata
I20260808 02:17:49.969990 4202 log.cc:1196] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/6a679ec659aa40748fda5541963c017b
I20260808 02:17:49.970350 4202 ts_tablet_manager.cc:1939] T 6a679ec659aa40748fda5541963c017b P 7928cf797acd409e856c0cca9cd44b20: Deleting consensus metadata
I20260808 02:17:49.970507 3489 catalog_manager.cc:4841] TS 79e074e5ab2346a5a005baf24b353a81 (127.28.177.130:35847): tablet 6a679ec659aa40748fda5541963c017b (table test-table [id=047c70bd0ec5435d8da3f4b31b7b2467]) successfully deleted
I20260808 02:17:49.970808 3487 catalog_manager.cc:4841] TS 7928cf797acd409e856c0cca9cd44b20 (127.28.177.131:40533): tablet 6a679ec659aa40748fda5541963c017b (table test-table [id=047c70bd0ec5435d8da3f4b31b7b2467]) successfully deleted
W20260808 02:17:49.972766 3490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:43167 (error 108)
I20260808 02:17:49.973111 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 3785
I20260808 02:17:49.973335 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:49.973195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:43167 (local address 127.0.0.1:49072)
0808 02:17:49.973250 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:49.973293 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 02:17:49.973719 3490 catalog_manager.cc:4584] TS 4ed4fd9ac54d403f814dd0af5657c2c9 (127.28.177.129:43167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a679ec659aa40748fda5541963c017b: Network error: Client connection negotiation failed: client connection to 127.28.177.129:43167: connect: Connection refused (error 111)
I20260808 02:17:49.985189 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 3969
I20260808 02:17:49.991348 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 3466
2026-08-08T02:17:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1087 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 02:17:50.017537 29382 mini_oidc.cc:152] Starting JWKS server
I20260808 02:17:50.018688 29382 webserver.cc:460] Webserver started at https://127.0.0.1:38907/ using document root <none> and password file <none>
I20260808 02:17:50.018807 29382 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:17:50.018990 29382 webserver.cc:460] Webserver started at http://127.0.0.1:43639/ using document root <none> and password file <none>
2026-08-08T02:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:50Z Disabled control of system clock
I20260808 02:17:50.028244 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:35213
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:35213
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38907/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/testchainca.cert with env {}
W20260808 02:17:50.112779 4214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:50.113092 4214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:50.113137 4214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:50.114400 4214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:50.114486 4214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:50.114511 4214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:50.114532 4214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:50.114558 4214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:17:50.114590 4214 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:17:50.114611 4214 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38907/jwks.json
I20260808 02:17:50.116279 4214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:35213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:35213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38907/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:50.116597 4214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:50.116854 4214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:50.122473 4224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.122460 4221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.122460 4220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:50.122725 4214 server_base.cc:1034] running on GCE node
I20260808 02:17:50.123070 4214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:50.123384 4214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:50.124573 4214 hybrid_clock.cc:648] HybridClock initialized: now 1786155470124549 us; error 63 us; skew 500 ppm
I20260808 02:17:50.126117 4214 webserver.cc:460] Webserver started at http://127.28.177.190:36877/ using document root <none> and password file <none>
I20260808 02:17:50.126374 4214 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:50.126446 4214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:50.126629 4214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:50.127739 4214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "89cd41d90197423fabee69726c00cb20"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "df6efba8474063fc603ec8acfb3d1c0b"
server_key_iv: "e7ca1cca53389be378e3377da68ec94d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.128139 4214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "89cd41d90197423fabee69726c00cb20"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "df6efba8474063fc603ec8acfb3d1c0b"
server_key_iv: "e7ca1cca53389be378e3377da68ec94d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.129667 4214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 02:17:50.130551 4232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:50.130820 4214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:50.130939 4214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "89cd41d90197423fabee69726c00cb20"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "df6efba8474063fc603ec8acfb3d1c0b"
server_key_iv: "e7ca1cca53389be378e3377da68ec94d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.131021 4214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:50.143407 4214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:50.143780 4214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:50.143944 4214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:50.148686 4214 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:35213
I20260808 02:17:50.148768 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:35213 every 8 connection(s)
I20260808 02:17:50.149199 4214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:50.150365 4302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:50.153069 4302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20: Bootstrap starting.
I20260808 02:17:50.153097 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 4214
I20260808 02:17:50.153235 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:50.153426 29382 external_mini_cluster.cc:1428] Setting key f544d1826d6a49d64a14e286d1173621
I20260808 02:17:50.153927 4302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:50.154371 4302 log.cc:824] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:50.155248 4302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20: No bootstrap required, opened a new log
I20260808 02:17:50.157861 4302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cd41d90197423fabee69726c00cb20" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35213 } }
I20260808 02:17:50.158073 4302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:50.158114 4302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cd41d90197423fabee69726c00cb20, State: Initialized, Role: FOLLOWER
I20260808 02:17:50.158249 4302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cd41d90197423fabee69726c00cb20" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35213 } }
I20260808 02:17:50.158327 4302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:50.158360 4302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:50.158411 4302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:50.159247 4302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cd41d90197423fabee69726c00cb20" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35213 } }
I20260808 02:17:50.159376 4302 leader_election.cc:302] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89cd41d90197423fabee69726c00cb20; no voters:
I20260808 02:17:50.159576 4302 leader_election.cc:288] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:50.159615 4308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:50.159823 4308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [term 1 LEADER]: Becoming Leader. State: Replica: 89cd41d90197423fabee69726c00cb20, State: Running, Role: LEADER
I20260808 02:17:50.159925 4302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:50.159972 4308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cd41d90197423fabee69726c00cb20" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35213 } }
I20260808 02:17:50.160420 4310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89cd41d90197423fabee69726c00cb20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cd41d90197423fabee69726c00cb20" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35213 } } }
I20260808 02:17:50.160501 4310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:50.160866 4316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:50.160367 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89cd41d90197423fabee69726c00cb20. Latest consensus state: current_term: 1 leader_uuid: "89cd41d90197423fabee69726c00cb20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cd41d90197423fabee69726c00cb20" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35213 } } }
I20260808 02:17:50.161432 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:50.160890 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.154020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.28.177.190:35213)
0808 02:17:50.154311 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.154321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.154370 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:17:50.155011 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.155021 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.155120 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.155295 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:50.155546 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.157145 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.157947 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.158637 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.158775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.159512 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:50.159518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:50.159518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:50.159761 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:50.159810 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:50.159827 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:50.160233 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:50.160270 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:17:50.160361 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260808 02:17:50.161932 4316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:50.162596 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:17:50.163815 4316 catalog_manager.cc:1347] Generated new cluster ID: 66c170b2f97b4cf0907262de50d91363
I20260808 02:17:50.163873 4316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:50.175768 4316 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:50.176518 4316 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:50.185631 4316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89cd41d90197423fabee69726c00cb20: Generated new TSK 0
I20260808 02:17:50.185870 4316 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:17:50.216771 4344 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 02:17:50.216964 4344 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:17:50.217067 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.193729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.28.177.190:35213)
0808 02:17:50.193802 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.193805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.193977 (+ 172us) server_negotiation.cc:407] Connection header received
0808 02:17:50.194100 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.194101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.194113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.194185 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:17:50.194349 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.194869 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.195678 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.195894 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.195986 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.196218 (+ 232us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:17:50.216732 (+ 20514us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:17:50.216988 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46994: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 02:17:50.217180 4306 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46994: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 02:17:50.217742 4343 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:50.193650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35213 (local address 127.0.0.1:46994)
0808 02:17:50.193891 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:50.193901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:50.193983 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:50.194196 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:50.194197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:50.194199 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:17:50.194270 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:50.194278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.194970 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.194971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:50.195586 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:50.195588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.196000 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.196002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:50.196065 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.196093 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:17:50.217116 (+ 21023us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:17:50.217388 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.177.190:35213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.177.190:35213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 02:17:50.218222 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 4214
2026-08-08T02:17:50Z chronyd exiting
I20260808 02:17:50.263546 29382 test_util.cc:175] -----------------------------------------------
I20260808 02:17:50.263708 29382 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155444117209-29382-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 02:17:50.264542 29382 mini_oidc.cc:152] Starting JWKS server
I20260808 02:17:50.265352 29382 webserver.cc:460] Webserver started at https://127.0.0.1:35929/ using document root <none> and password file <none>
I20260808 02:17:50.265408 29382 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:17:50.265587 29382 webserver.cc:460] Webserver started at http://127.0.0.1:42333/ using document root <none> and password file <none>
2026-08-08T02:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:50Z Disabled control of system clock
I20260808 02:17:50.275481 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36149
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:36149
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35929/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/testchainca.cert with env {}
W20260808 02:17:50.359130 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:50.359351 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:50.359376 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:50.360584 4350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:50.360627 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:50.360641 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:50.360656 4350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:50.360702 4350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:17:50.360733 4350 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:17:50.360749 4350 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35929/jwks.json
I20260808 02:17:50.362358 4350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:36149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35929/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:50.362639 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:50.362896 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:50.368261 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.368234 4358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.368294 4356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:50.368659 4350 server_base.cc:1034] running on GCE node
I20260808 02:17:50.369004 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:50.369280 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:50.370430 4350 hybrid_clock.cc:648] HybridClock initialized: now 1786155470370417 us; error 38 us; skew 500 ppm
I20260808 02:17:50.371803 4350 webserver.cc:460] Webserver started at http://127.28.177.190:46593/ using document root <none> and password file <none>
I20260808 02:17:50.372051 4350 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:50.372097 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:50.372254 4350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:50.373430 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "688275d223894489b8e12d2e34432b42"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "beb13717ad66712d313a129415010811"
server_key_iv: "dfb14a2bdc113c64639fc74c0a59ec26"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.373826 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "688275d223894489b8e12d2e34432b42"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "beb13717ad66712d313a129415010811"
server_key_iv: "dfb14a2bdc113c64639fc74c0a59ec26"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.375286 4350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:50.376145 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:50.376338 4350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:50.376462 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "688275d223894489b8e12d2e34432b42"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "beb13717ad66712d313a129415010811"
server_key_iv: "dfb14a2bdc113c64639fc74c0a59ec26"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.376542 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:50.410727 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:50.411093 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:50.411269 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:50.415745 4350 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:36149
I20260808 02:17:50.415796 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:36149 every 8 connection(s)
I20260808 02:17:50.416201 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:50.417069 4440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:50.419991 4440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42: Bootstrap starting.
I20260808 02:17:50.420514 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 4350
I20260808 02:17:50.420631 4440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:50.420636 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:50.420902 29382 external_mini_cluster.cc:1428] Setting key 949b1d3d874c5b071b1038be3f2b223b
I20260808 02:17:50.421042 4440 log.cc:824] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:50.421918 4440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42: No bootstrap required, opened a new log
I20260808 02:17:50.424206 4440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688275d223894489b8e12d2e34432b42" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36149 } }
I20260808 02:17:50.424387 4440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:50.424418 4440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688275d223894489b8e12d2e34432b42, State: Initialized, Role: FOLLOWER
I20260808 02:17:50.424540 4440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688275d223894489b8e12d2e34432b42" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36149 } }
I20260808 02:17:50.424615 4440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:50.424667 4440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:50.424732 4440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:50.425375 4440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688275d223894489b8e12d2e34432b42" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36149 } }
I20260808 02:17:50.425509 4440 leader_election.cc:302] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688275d223894489b8e12d2e34432b42; no voters:
I20260808 02:17:50.425712 4440 leader_election.cc:288] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:50.425779 4450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:50.425920 4450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [term 1 LEADER]: Becoming Leader. State: Replica: 688275d223894489b8e12d2e34432b42, State: Running, Role: LEADER
I20260808 02:17:50.426029 4440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:50.426017 4450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688275d223894489b8e12d2e34432b42" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36149 } }
I20260808 02:17:50.426548 4451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688275d223894489b8e12d2e34432b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688275d223894489b8e12d2e34432b42" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36149 } } }
I20260808 02:17:50.426649 4451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:50.426816 4452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688275d223894489b8e12d2e34432b42. Latest consensus state: current_term: 1 leader_uuid: "688275d223894489b8e12d2e34432b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688275d223894489b8e12d2e34432b42" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36149 } } }
I20260808 02:17:50.426887 4452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:50.427218 4459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:50.427512 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.421487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.28.177.190:36149)
0808 02:17:50.421673 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.421681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.421715 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:17:50.422222 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.422230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.422290 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.422411 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:50.422663 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.423559 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.424707 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.425375 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.425521 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.426341 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:50.426345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:50.426345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:50.426590 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:50.426635 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:50.426651 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:50.426717 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:50.427161 (+ 444us) server_negotiation.cc:299] Negotiation successful
0808 02:17:50.427253 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260808 02:17:50.427717 4459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:50.428905 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:17:50.430011 4459 catalog_manager.cc:1347] Generated new cluster ID: 37461b15105543fab3e928df7c59ab23
I20260808 02:17:50.430068 4459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:50.445879 4459 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:50.446681 4459 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:50.456710 4459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688275d223894489b8e12d2e34432b42: Generated new TSK 0
I20260808 02:17:50.456995 4459 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:17:50.488878 4485 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 02:17:50.488984 4485 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:17:50.489326 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.466873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.28.177.190:36149)
0808 02:17:50.466943 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.466945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.467070 (+ 125us) server_negotiation.cc:407] Connection header received
0808 02:17:50.467157 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.467158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.467169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.467233 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:17:50.467389 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.467900 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.469097 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.469303 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.469365 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.469534 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:17:50.489080 (+ 19546us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:17:50.489268 (+ 188us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 02:17:50.489403 4449 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 02:17:50.489455 4482 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:50.466803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36149 (local address 127.0.0.1:49812)
0808 02:17:50.466992 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:50.467005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:50.467077 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:50.467254 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:50.467255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:50.467258 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:17:50.467323 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:50.467326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.467991 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.467992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:50.469016 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:50.469019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.469379 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.469380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:50.469439 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.469445 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:17:50.489227 (+ 19782us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:17:50.489353 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.177.190:36149: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260808 02:17:50.489794 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 4350
2026-08-08T02:17:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (272 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 02:17:50.537731 29382 mini_oidc.cc:152] Starting JWKS server
I20260808 02:17:50.538702 29382 webserver.cc:460] Webserver started at https://127.0.0.1:33977/ using document root <none> and password file <none>
I20260808 02:17:50.538750 29382 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:17:50.538878 29382 webserver.cc:460] Webserver started at http://127.0.0.1:36547/ using document root <none> and password file <none>
2026-08-08T02:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:50Z Disabled control of system clock
I20260808 02:17:50.549090 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:46465
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:46465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:17:50.633893 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:50.634140 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:50.634164 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:50.635293 4492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:50.635329 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:50.635342 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:50.635355 4492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:50.635367 4492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:17:50.635380 4492 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:17:50.635391 4492 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33977/jwks.json
I20260808 02:17:50.636713 4492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:46465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:50.636929 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:50.637185 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:50.642241 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:50.642361 4492 server_base.cc:1034] running on GCE node
W20260808 02:17:50.642254 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.642272 4501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:50.642786 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:50.643090 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:50.644289 4492 hybrid_clock.cc:648] HybridClock initialized: now 1786155470644268 us; error 52 us; skew 500 ppm
I20260808 02:17:50.645619 4492 webserver.cc:460] Webserver started at http://127.28.177.190:39335/ using document root <none> and password file <none>
I20260808 02:17:50.645887 4492 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:50.645958 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:50.646114 4492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:50.647251 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "0ebe3c1143ee7be081c3381b44523084"
server_key_iv: "00c83429d08648bab6db338e8668575d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.647631 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "0ebe3c1143ee7be081c3381b44523084"
server_key_iv: "00c83429d08648bab6db338e8668575d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.649071 4492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:50.649849 4511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:50.650017 4492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:50.650133 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "0ebe3c1143ee7be081c3381b44523084"
server_key_iv: "00c83429d08648bab6db338e8668575d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.650211 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:50.670310 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:50.670673 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:50.670841 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:50.675122 4492 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:46465
I20260808 02:17:50.675197 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:46465 every 8 connection(s)
I20260808 02:17:50.675661 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:50.676517 4582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:50.679397 4582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa: Bootstrap starting.
I20260808 02:17:50.680004 4582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:50.680323 4582 log.cc:824] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:50.681202 4582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa: No bootstrap required, opened a new log
I20260808 02:17:50.682796 4582 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46465 } }
I20260808 02:17:50.682933 4582 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:50.682955 4582 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbcd5a9cdff4d33ad967e4ca1b422aa, State: Initialized, Role: FOLLOWER
I20260808 02:17:50.683148 4582 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46465 } }
I20260808 02:17:50.683223 4582 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:50.683245 4582 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:50.683297 4582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:50.683858 4582 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46465 } }
I20260808 02:17:50.683943 4582 leader_election.cc:302] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbbcd5a9cdff4d33ad967e4ca1b422aa; no voters:
I20260808 02:17:50.684103 4582 leader_election.cc:288] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:50.684149 4587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:50.684190 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 4492
I20260808 02:17:50.684300 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:50.684314 4587 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [term 1 LEADER]: Becoming Leader. State: Replica: dbbcd5a9cdff4d33ad967e4ca1b422aa, State: Running, Role: LEADER
I20260808 02:17:50.684368 4582 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:50.684412 4587 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46465 } }
I20260808 02:17:50.684516 29382 external_mini_cluster.cc:1428] Setting key 2494163b69c451caabe912316e781aae
I20260808 02:17:50.685493 4589 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46465 } } }
I20260808 02:17:50.685590 4589 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:50.686062 4590 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbbcd5a9cdff4d33ad967e4ca1b422aa. Latest consensus state: current_term: 1 leader_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbbcd5a9cdff4d33ad967e4ca1b422aa" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 46465 } } }
I20260808 02:17:50.686267 4590 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa [sys.catalog]: This master's current role is: LEADER
W20260808 02:17:50.686478 4609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:17:50.686544 4609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:17:50.686594 4608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:50.686758 4608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:50.688282 4608 catalog_manager.cc:1347] Generated new cluster ID: e89dbb6c02214271b2b6c346db2a5ae6
I20260808 02:17:50.688339 4608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:50.690939 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.685007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.28.177.190:46465)
0808 02:17:50.685908 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.685919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.685952 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:17:50.686069 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.686076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.686142 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.686621 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:50.686747 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.687963 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.688945 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.689378 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.689512 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.690261 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:50.690265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:50.690266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:50.690447 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:50.690480 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:50.690490 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:50.690566 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:50.690641 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:17:50.690728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":90,"threads_started":1}
I20260808 02:17:50.691905 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:17:50.722901 4608 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:50.723462 4608 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:50.729875 4608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbbcd5a9cdff4d33ad967e4ca1b422aa: Generated new TSK 0
I20260808 02:17:50.730085 4608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:17:50.742451 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.738629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.28.177.190:46465)
0808 02:17:50.738687 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.738689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.738790 (+ 101us) server_negotiation.cc:407] Connection header received
0808 02:17:50.738881 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.738882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.738904 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.739016 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:17:50.739183 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.739716 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.740817 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.741090 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.741146 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.741375 (+ 229us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:17:50.742215 (+ 840us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:17:50.742374 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260808 02:17:50.742518 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:50.738524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:46465 (local address 127.0.0.1:40516)
0808 02:17:50.738704 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:50.738717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:50.738793 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:50.739031 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:50.739032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:50.739034 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:17:50.739100 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:50.739102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.739797 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.739799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:50.740728 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:50.740730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.741181 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.741183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:50.741251 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.741259 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:17:50.742318 (+ 1059us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:17:50.742420 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.177.190:46465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
W20260808 02:17:50.742523 4600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260808 02:17:50.742835 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 4492
E20260808 02:17:50.766034 4622 webserver.cc:533] Webserver: sslize failed:
E20260808 02:17:50.766129 4622 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T02:17:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (256 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T02:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:50Z Disabled control of system clock
I20260808 02:17:50.803195 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:38101
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:38101
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/testchainca.cert with env {}
W20260808 02:17:50.910392 4628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:50.910768 4628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:50.910833 4628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:50.913506 4628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:50.913595 4628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:50.913623 4628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:50.913653 4628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:50.913715 4628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:17:50.913745 4628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:17:50.913781 4628 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 02:17:50.917228 4628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:38101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:50.917634 4628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:50.917994 4628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:50.924363 4634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.924371 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:50.924371 4635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:50.924708 4628 server_base.cc:1034] running on GCE node
I20260808 02:17:50.925076 4628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:50.925604 4628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:50.926838 4628 hybrid_clock.cc:648] HybridClock initialized: now 1786155470926807 us; error 51 us; skew 500 ppm
I20260808 02:17:50.928357 4628 webserver.cc:460] Webserver started at http://127.28.177.190:33663/ using document root <none> and password file <none>
I20260808 02:17:50.928620 4628 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:50.928694 4628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:50.928897 4628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:50.930025 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "4564007719c7453dad18826ce7d027f0"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "e1c99e645771dc9d73ea50dd4cf7b83f"
server_key_iv: "badb36d0fd62ecbef980c1a321e3818d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.930420 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "4564007719c7453dad18826ce7d027f0"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "e1c99e645771dc9d73ea50dd4cf7b83f"
server_key_iv: "badb36d0fd62ecbef980c1a321e3818d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.931833 4628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:50.932760 4644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:50.933085 4628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:50.933223 4628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "4564007719c7453dad18826ce7d027f0"
format_stamp: "Formatted at 2026-08-08 02:17:50 on dist-test-slave-3h5h"
server_key: "e1c99e645771dc9d73ea50dd4cf7b83f"
server_key_iv: "badb36d0fd62ecbef980c1a321e3818d"
server_key_version: "encryptionkey@0"
I20260808 02:17:50.933305 4628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:50.967702 4628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:50.968091 4628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:50.968271 4628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:50.972968 4628 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:38101
I20260808 02:17:50.973039 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:38101 every 8 connection(s)
I20260808 02:17:50.973434 4628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:50.974575 4725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:50.977334 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0: Bootstrap starting.
I20260808 02:17:50.977936 4725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:50.978209 4725 log.cc:824] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:50.978740 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 4628
I20260808 02:17:50.978883 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:50.979096 29382 external_mini_cluster.cc:1428] Setting key cbe3b44e7d5bf6b759c07af766dd9215
I20260808 02:17:50.979197 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0: No bootstrap required, opened a new log
I20260808 02:17:50.981971 4725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4564007719c7453dad18826ce7d027f0" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38101 } }
I20260808 02:17:50.982146 4725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:50.982196 4725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4564007719c7453dad18826ce7d027f0, State: Initialized, Role: FOLLOWER
I20260808 02:17:50.982375 4725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4564007719c7453dad18826ce7d027f0" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38101 } }
I20260808 02:17:50.982487 4725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:50.982565 4725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:50.982625 4725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:50.983456 4725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4564007719c7453dad18826ce7d027f0" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38101 } }
I20260808 02:17:50.983618 4725 leader_election.cc:302] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4564007719c7453dad18826ce7d027f0; no voters:
I20260808 02:17:50.983839 4725 leader_election.cc:288] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:50.983861 4733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:50.983976 4733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [term 1 LEADER]: Becoming Leader. State: Replica: 4564007719c7453dad18826ce7d027f0, State: Running, Role: LEADER
I20260808 02:17:50.984130 4725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:50.984130 4733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4564007719c7453dad18826ce7d027f0" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38101 } }
I20260808 02:17:50.985631 4735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4564007719c7453dad18826ce7d027f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4564007719c7453dad18826ce7d027f0" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38101 } } }
I20260808 02:17:50.985813 4735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [sys.catalog]: This master's current role is: LEADER
W20260808 02:17:50.986217 4750 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 02:17:50.986236 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:50.979770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.28.177.190:38101)
0808 02:17:50.980103 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:50.980111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:50.980142 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:17:50.980655 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:50.980662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:50.980720 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:50.980898 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:50.981284 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.982347 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.983484 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:50.984304 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:50.984440 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:50.985293 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:50.985297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:50.985297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:17:50.985581 (+ 284us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:50.985627 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:17:50.985644 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:50.985717 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:50.985938 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 02:17:50.986018 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
W20260808 02:17:50.986297 4750 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:17:50.986598 4734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4564007719c7453dad18826ce7d027f0. Latest consensus state: current_term: 1 leader_uuid: "4564007719c7453dad18826ce7d027f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4564007719c7453dad18826ce7d027f0" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38101 } } }
I20260808 02:17:50.986677 4734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:50.986779 4754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:50.986974 4754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:50.987329 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:17:50.988701 4754 catalog_manager.cc:1347] Generated new cluster ID: 9090eb396c6f4dd7a572bf5ede7c2318
I20260808 02:17:50.988761 4754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:51.002887 4754 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:51.003708 4754 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:51.010241 4754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4564007719c7453dad18826ce7d027f0: Generated new TSK 0
I20260808 02:17:51.010515 4754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:17:51.056938 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:51.015603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.28.177.190:38101)
0808 02:17:51.015656 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:51.015658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:51.015766 (+ 108us) server_negotiation.cc:407] Connection header received
0808 02:17:51.015837 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:51.015838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:51.015853 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:51.015927 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:17:51.016053 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:51.016889 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:51.017664 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:51.018002 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:51.018075 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:51.018243 (+ 168us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:17:51.056530 (+ 38287us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:17:51.056819 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54366: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":28}
W20260808 02:17:51.057032 4731 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54366: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 02:17:51.057035 4764 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:51.015526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38101 (local address 127.0.0.1:54366)
0808 02:17:51.015684 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:51.015695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:51.015761 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:51.015932 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:51.015933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:51.015934 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:17:51.015986 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:51.015988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:51.016976 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:51.016977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:51.017575 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:51.017576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:51.018090 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:51.018091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:51.018133 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:51.018137 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:17:51.056759 (+ 38622us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:17:51.056923 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.177.190:38101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260808 02:17:51.057394 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 4628
2026-08-08T02:17:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (287 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (93 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:51 dist-test-slave-3h5h krb5kdc[4784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:51 dist-test-slave-3h5h krb5kdc[4784](info): set up 2 sockets
Aug 08 02:17:51 dist-test-slave-3h5h krb5kdc[4784](info): commencing operation
krb5kdc: starting...
W20260808 02:17:52.575767 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:52 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:17:52.712539 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37991
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:52.819020 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:52.819423 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:52.819466 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:52.821269 4801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:52.821329 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:52.821357 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:52.821381 4801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:52.821399 4801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:52.823486 4801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:37991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:52.823818 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:52.824040 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:52.831096 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:52.831096 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:52.831432 4801 server_base.cc:1034] running on GCE node
W20260808 02:17:52.831267 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:52.831954 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:52.832391 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:52.833606 4801 hybrid_clock.cc:648] HybridClock initialized: now 1786155472833586 us; error 51 us; skew 500 ppm
Aug 08 02:17:52 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:52.837680 4801 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:17:52.838627 4801 webserver.cc:460] Webserver started at http://127.28.177.190:37083/ using document root <none> and password file <none>
I20260808 02:17:52.839025 4801 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:52.839136 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:52.839411 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:52.841310 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "cf6f385e486543199698d5e331137ae9"
format_stamp: "Formatted at 2026-08-08 02:17:52 on dist-test-slave-3h5h"
server_key: "b99625901acdd420825133e59dc1979a"
server_key_iv: "63f26784b8221dcdb1b923778d0d85d5"
server_key_version: "encryptionkey@0"
I20260808 02:17:52.842025 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "cf6f385e486543199698d5e331137ae9"
format_stamp: "Formatted at 2026-08-08 02:17:52 on dist-test-slave-3h5h"
server_key: "b99625901acdd420825133e59dc1979a"
server_key_iv: "63f26784b8221dcdb1b923778d0d85d5"
server_key_version: "encryptionkey@0"
I20260808 02:17:52.844461 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:17:52.845819 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:52.846136 4801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:52.846290 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "cf6f385e486543199698d5e331137ae9"
format_stamp: "Formatted at 2026-08-08 02:17:52 on dist-test-slave-3h5h"
server_key: "b99625901acdd420825133e59dc1979a"
server_key_iv: "63f26784b8221dcdb1b923778d0d85d5"
server_key_version: "encryptionkey@0"
I20260808 02:17:52.846379 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:52.890429 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:52.894066 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:52.894331 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:52.900457 4801 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:37991
I20260808 02:17:52.900517 4894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:37991 every 8 connection(s)
I20260808 02:17:52.901113 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:52.902453 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:52.905440 4895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9: Bootstrap starting.
I20260808 02:17:52.906215 4895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:52.906579 4895 log.cc:824] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:52.907688 4895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9: No bootstrap required, opened a new log
I20260808 02:17:52.909752 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 4801
I20260808 02:17:52.909909 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:52.910142 29382 external_mini_cluster.cc:1428] Setting key 93bc0fba30e7fe0aa87b19cfb7ebbdb0
I20260808 02:17:52.910133 4895 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6f385e486543199698d5e331137ae9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37991 } }
I20260808 02:17:52.910416 4895 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:52.910502 4895 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6f385e486543199698d5e331137ae9, State: Initialized, Role: FOLLOWER
I20260808 02:17:52.910769 4895 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6f385e486543199698d5e331137ae9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37991 } }
I20260808 02:17:52.911005 4895 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:52.911116 4895 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:52.911259 4895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:52.912956 4895 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6f385e486543199698d5e331137ae9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37991 } }
I20260808 02:17:52.913185 4895 leader_election.cc:302] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6f385e486543199698d5e331137ae9; no voters:
I20260808 02:17:52.913398 4895 leader_election.cc:288] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:52.913529 4902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:52.913800 4895 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:52.913857 4902 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [term 1 LEADER]: Becoming Leader. State: Replica: cf6f385e486543199698d5e331137ae9, State: Running, Role: LEADER
I20260808 02:17:52.914248 4902 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6f385e486543199698d5e331137ae9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37991 } }
I20260808 02:17:52.915408 4904 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6f385e486543199698d5e331137ae9. Latest consensus state: current_term: 1 leader_uuid: "cf6f385e486543199698d5e331137ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6f385e486543199698d5e331137ae9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37991 } } }
I20260808 02:17:52.915654 4904 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:52.916133 4903 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6f385e486543199698d5e331137ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6f385e486543199698d5e331137ae9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37991 } } }
I20260808 02:17:52.916041 4919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:52.916678 4903 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:52.917268 4919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:52.919770 4919 catalog_manager.cc:1347] Generated new cluster ID: de35eeabe8164f76bb5a0932371de52c
I20260808 02:17:52.919847 4919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:17:52 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:52.929749 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:52.910905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.28.177.190:37991)
0808 02:17:52.911243 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:52.911257 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:52.911314 (+ 57us) server_negotiation.cc:407] Connection header received
0808 02:17:52.911534 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:52.911549 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:52.911632 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:52.911880 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:52.913123 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:52.915082 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:52.916859 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:52.917625 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:52.917789 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:52.922933 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:52.922938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:52.922939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:52.922958 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:52.926148 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:52.926808 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:52.926813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:52.926813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:52.926996 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:52.927342 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:52.927343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:52.927343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:52.928000 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:52.928297 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:52.929096 (+ 799us) server_negotiation.cc:299] Negotiation successful
0808 02:17:52.929374 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":171,"threads_started":1}
I20260808 02:17:52.931614 4919 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:52.932476 4919 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:52.953111 4919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6f385e486543199698d5e331137ae9: Generated new TSK 0
I20260808 02:17:52.953467 4919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:17:53.009253 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37991
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:53.113512 4928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:53.113736 4928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:53.113761 4928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:53.115064 4928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:53.115113 4928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:53.115139 4928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:53.116771 4928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:53.117081 4928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:53.117336 4928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:53.122743 4940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:53.122777 4928 server_base.cc:1034] running on GCE node
W20260808 02:17:53.122733 4942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:53.122716 4938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:53.123220 4928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:53.123466 4928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:53.124663 4928 hybrid_clock.cc:648] HybridClock initialized: now 1786155473124639 us; error 54 us; skew 500 ppm
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.127470 4928 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:17:53.128046 4928 webserver.cc:460] Webserver started at http://127.28.177.129:36647/ using document root <none> and password file <none>
I20260808 02:17:53.128396 4928 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:53.128480 4928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:53.128656 4928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:53.129913 4928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "5c3fdaf64d9f4972b0e490f6fd1317eb"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "e29f54cf63f305843277f4e305a5cd74"
server_key_iv: "a87f1dcd5e079e980074ba2b2ddd564e"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.130343 4928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "5c3fdaf64d9f4972b0e490f6fd1317eb"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "e29f54cf63f305843277f4e305a5cd74"
server_key_iv: "a87f1dcd5e079e980074ba2b2ddd564e"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.132296 4928 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:17:53.133467 4951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.133756 4928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:53.133885 4928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "5c3fdaf64d9f4972b0e490f6fd1317eb"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "e29f54cf63f305843277f4e305a5cd74"
server_key_iv: "a87f1dcd5e079e980074ba2b2ddd564e"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.133978 4928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:53.144461 4928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:53.147378 4928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:53.147583 4928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:53.147847 4928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:53.148211 4928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:53.148269 4928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.148308 4928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:53.148357 4928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.155215 4928 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45435
I20260808 02:17:53.155264 5113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45435 every 8 connection(s)
I20260808 02:17:53.155616 4928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:53.160171 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 4928
I20260808 02:17:53.160323 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:53.160585 29382 external_mini_cluster.cc:1428] Setting key c8b57ee549d92fae185ddec92f8fe75e
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:53.167057 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.156437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:38793 (local address 127.28.177.190:37991)
0808 02:17:53.156523 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.156525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.156740 (+ 215us) server_negotiation.cc:407] Connection header received
0808 02:17:53.157835 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.157836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.157858 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.157941 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.158631 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.159215 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.160560 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.160891 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.161018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.163713 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:53.163715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:53.163715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:53.163724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:53.165424 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.165969 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.165970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.165970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.166041 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.166316 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.166317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.166317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.166444 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:53.166520 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:53.166833 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 02:17:53.166942 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:17:53.167223 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.156401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37991 (local address 127.28.177.129:38793)
0808 02:17:53.156583 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:53.156630 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:53.157700 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:53.158019 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:53.158024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:53.158289 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:53.158490 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:53.158493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.159344 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.159345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:53.160450 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:53.160454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.161034 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.161035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:53.161183 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.161800 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:53.161809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:53.163585 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:53.165543 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:53.165551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:53.165553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:53.165884 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:53.166113 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:53.166115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:53.166115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:53.166238 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:53.166522 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:53.166530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:53.166686 (+ 156us) client_negotiation.cc:769] Sending connection context
0808 02:17:53.166795 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:17:53.166969 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260808 02:17:53.167747 5114 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37991
I20260808 02:17:53.167883 5114 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:53.168094 5114 heartbeater.cc:511] Master 127.28.177.190:37991 requested a full tablet report, sending...
I20260808 02:17:53.168681 4841 ts_manager.cc:194] Registered new tserver with Master: 5c3fdaf64d9f4972b0e490f6fd1317eb (127.28.177.129:45435)
I20260808 02:17:53.169480 4841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:38793
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:17:53.190666 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37991
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:53.275400 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:53.275619 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:53.275643 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:53.276816 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:53.276880 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:53.276906 5124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:53.278411 5124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:53.278666 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:53.278903 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:53.284323 5133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:53.284282 5134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:53.284260 5141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:53.284559 5124 server_base.cc:1034] running on GCE node
I20260808 02:17:53.284765 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:53.285049 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:53.286221 5124 hybrid_clock.cc:648] HybridClock initialized: now 1786155473286231 us; error 53 us; skew 500 ppm
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.288726 5124 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:17:53.289234 5124 webserver.cc:460] Webserver started at http://127.28.177.130:46475/ using document root <none> and password file <none>
I20260808 02:17:53.289464 5124 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:53.289534 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:53.289690 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:53.290776 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "ea5f3438c88c4092afae9308bc227978"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "ed6caefb824ffb739fbfa43a09e39bfc"
server_key_iv: "e2e5efe49b68e53739b04e9976f577e0"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.291164 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "ea5f3438c88c4092afae9308bc227978"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "ed6caefb824ffb739fbfa43a09e39bfc"
server_key_iv: "e2e5efe49b68e53739b04e9976f577e0"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.292557 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:53.293444 5150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.293617 5124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:53.293741 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "ea5f3438c88c4092afae9308bc227978"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "ed6caefb824ffb739fbfa43a09e39bfc"
server_key_iv: "e2e5efe49b68e53739b04e9976f577e0"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.293828 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:53.307955 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:53.310385 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:53.310583 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:53.310873 5124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:53.311250 5124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:53.311313 5124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.311362 5124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:53.311393 5124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.318813 5124 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:36717
I20260808 02:17:53.318843 5315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:36717 every 8 connection(s)
I20260808 02:17:53.319267 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:53.327221 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 5124
I20260808 02:17:53.327327 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:53.327608 29382 external_mini_cluster.cc:1428] Setting key c74684d1a865d159b5958e1023c9b1d6
I20260808 02:17:53.330224 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.320326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:50959 (local address 127.28.177.190:37991)
0808 02:17:53.320403 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.320405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.320416 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:53.321514 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.321515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.321529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.321595 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.322239 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.322762 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.323798 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.324019 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.324101 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.326533 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:53.326534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:53.326535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:53.326543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:53.328595 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.329091 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.329093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.329093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.329170 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.329434 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.329435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.329436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.329614 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:53.329688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:53.329951 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:17:53.330063 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:17:53.330241 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.320000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37991 (local address 127.28.177.130:50959)
0808 02:17:53.320183 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:53.320228 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:53.321396 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:53.321675 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:53.321678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:53.321941 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:53.322140 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:53.322144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.322866 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.322867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:53.323700 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:53.323703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.324110 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.324111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:53.324234 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.324742 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:53.324750 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:53.326429 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:53.328697 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:53.328704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:53.328706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:53.329018 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:53.329246 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:53.329247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:53.329247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:53.329352 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:53.329679 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:53.329684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:53.329805 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 02:17:53.329886 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:17:53.330011 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260808 02:17:53.330881 5316 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37991
I20260808 02:17:53.331012 5316 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:53.331203 5316 heartbeater.cc:511] Master 127.28.177.190:37991 requested a full tablet report, sending...
I20260808 02:17:53.331681 4841 ts_manager.cc:194] Registered new tserver with Master: ea5f3438c88c4092afae9308bc227978 (127.28.177.130:36717)
I20260808 02:17:53.332072 4841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:50959
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:17:53.361987 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37991
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:53.448055 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:53.448300 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:53.448338 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:53.449519 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:53.449597 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:53.449644 5321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:53.451236 5321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:53.451508 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:53.451784 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:53.457566 5330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:53.457612 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:53.457726 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:53.457829 5321 server_base.cc:1034] running on GCE node
I20260808 02:17:53.458184 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:53.458460 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:53.459631 5321 hybrid_clock.cc:648] HybridClock initialized: now 1786155473459608 us; error 50 us; skew 500 ppm
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.462476 5321 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:17:53.462932 5321 webserver.cc:460] Webserver started at http://127.28.177.131:40951/ using document root <none> and password file <none>
I20260808 02:17:53.463186 5321 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:53.463235 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:53.463330 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:53.464342 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "df7bf1f1a9454f9283bbe20d58e42faf"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "526b301313fbab4f9c6d41dd79c48e69"
server_key_iv: "6e6ea25a1a922cb6cd561d022b961c18"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.464658 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "df7bf1f1a9454f9283bbe20d58e42faf"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "526b301313fbab4f9c6d41dd79c48e69"
server_key_iv: "6e6ea25a1a922cb6cd561d022b961c18"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.466142 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:53.466949 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.467175 5321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:53.467280 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "df7bf1f1a9454f9283bbe20d58e42faf"
format_stamp: "Formatted at 2026-08-08 02:17:53 on dist-test-slave-3h5h"
server_key: "526b301313fbab4f9c6d41dd79c48e69"
server_key_iv: "6e6ea25a1a922cb6cd561d022b961c18"
server_key_version: "encryptionkey@0"
I20260808 02:17:53.467348 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:53.477553 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:53.480101 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:53.480226 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:53.480504 5321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:53.480943 5321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:53.480988 5321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.481012 5321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:53.481026 5321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:53.487551 5321 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:42739
I20260808 02:17:53.487644 5507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:42739 every 8 connection(s)
I20260808 02:17:53.488013 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:17:53.498358 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 5321
I20260808 02:17:53.498534 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:53.498795 29382 external_mini_cluster.cc:1428] Setting key 78411a3939d18165b6476bf753eea443
I20260808 02:17:53.499096 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.488896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:39005 (local address 127.28.177.190:37991)
0808 02:17:53.488977 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.488979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.489189 (+ 210us) server_negotiation.cc:407] Connection header received
0808 02:17:53.490312 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.490314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.490327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.490395 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.491146 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.491638 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.492987 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.493262 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.493320 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.495804 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:53.495806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:53.495806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:53.495814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:53.497332 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.497880 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.497882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.497882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.497938 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.498303 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.498304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.498304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.498438 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:53.498509 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:53.498903 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 02:17:53.499003 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:17:53.499311 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.488824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37991 (local address 127.28.177.131:39005)
0808 02:17:53.489026 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:53.489072 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:53.490173 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:53.490485 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:53.490489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:53.490767 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:53.490970 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:53.490974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.491753 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.491755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:53.492862 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:53.492865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.493363 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.493364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:53.493486 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.494105 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:53.494115 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:53.495711 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:53.497432 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:53.497447 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:53.497448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:53.497810 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:53.498013 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:53.498013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:53.498014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:53.498218 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:53.498515 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:53.498541 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:53.498715 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 02:17:53.498842 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 02:17:53.499021 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260808 02:17:53.499774 5508 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37991
I20260808 02:17:53.499889 5508 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:53.500063 5508 heartbeater.cc:511] Master 127.28.177.190:37991 requested a full tablet report, sending...
I20260808 02:17:53.500451 4841 ts_manager.cc:194] Registered new tserver with Master: df7bf1f1a9454f9283bbe20d58e42faf (127.28.177.131:42739)
I20260808 02:17:53.500929 4841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:39005
I20260808 02:17:53.510967 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/test-admin.keytab.
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.531694 29382 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 02:17:53.533627 5519 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:53.532679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.28.177.129:45435)
0808 02:17:53.532879 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.532889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.532900 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:53.532950 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.532952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.532966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.533043 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.533507 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44848: BlockingRecv error: recv got EOF from 127.0.0.1:44848 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260808 02:17:53.533758 29382 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.534761 29382 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.535804 29382 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:53.544322 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.536369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.28.177.129:45435)
0808 02:17:53.536422 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.536425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.536542 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:17:53.536638 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.536638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.536648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.536702 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.537276 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.538166 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.538939 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.539266 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.539323 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.541560 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:53.541569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:53.541569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:53.541576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:53.543244 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.543595 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.543598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.543598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.543658 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.543835 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.543836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.543836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.543964 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:53.544075 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:53.544179 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 02:17:53.544243 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260808 02:17:53.546775 5519 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:53.546096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.28.177.129:45435)
0808 02:17:53.546165 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.546166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.546174 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:53.546239 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.546240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.546249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.546293 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.546734 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44874: BlockingRecv error: recv got EOF from 127.0.0.1:44874 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 02:17:53.546949 29382 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc)
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.548157 29382 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.549228 29382 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:53.557122 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.549762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.28.177.129:45435)
0808 02:17:53.549818 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.549820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.549874 (+ 54us) server_negotiation.cc:407] Connection header received
0808 02:17:53.549918 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.549918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.549928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.549974 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.550435 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.550895 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.551655 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.551839 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.551892 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.554429 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:53.554431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:53.554431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:53.554438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:53.555914 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.556340 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.556341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.556341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.556394 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.556600 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.556601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.556601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.556691 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:53.556761 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:53.556930 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:17:53.556996 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260808 02:17:53.559037 5519 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:17:53.558524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.28.177.129:45435)
0808 02:17:53.558584 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.558586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.558614 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:17:53.558672 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.558673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.558681 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.558722 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.558994 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44902: BlockingRecv error: recv got EOF from 127.0.0.1:44902 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 02:17:53.559198 29382 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155444117209-29382-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.560292 29382 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:53.561415 29382 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[4784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:17:53.569236 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:53.562132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.28.177.129:45435)
0808 02:17:53.562213 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:53.562215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:53.562223 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:53.562268 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:53.562269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:53.562276 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:53.562323 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:53.562691 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.563146 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.563907 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:53.564077 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:53.564125 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:53.566593 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:53.566594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:53.566595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:53.566601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:53.568027 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.568404 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.568405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.568405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.568458 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:53.568675 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:53.568676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:53.568676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:53.568764 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:53.568880 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:53.569023 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 02:17:53.569080 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:17:53.570399 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 4928
I20260808 02:17:53.583475 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 5124
I20260808 02:17:53.605836 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 5321
I20260808 02:17:53.615584 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 4801
2026-08-08T02:17:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2366 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[5540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[5540](info): set up 2 sockets
Aug 08 02:17:53 dist-test-slave-3h5h krb5kdc[5540](info): commencing operation
krb5kdc: starting...
W20260808 02:17:54.964568 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.324s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:17:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:17:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab.
Entry for principal oryx/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab.
I20260808 02:17:55.101238 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42235
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab
--principal=oryx/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:55.191087 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:55.191322 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:55.191347 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:55.192508 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:55.192548 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:55.192564 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:55.192577 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:55.192589 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:55.194229 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab
--principal=oryx/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:55.194451 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:55.194639 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:55.200266 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.200259 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.200254 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:55.200519 5560 server_base.cc:1034] running on GCE node
I20260808 02:17:55.200958 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:55.201269 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:55.202447 5560 hybrid_clock.cc:648] HybridClock initialized: now 1786155475202427 us; error 34 us; skew 500 ppm
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:55.205302 5560 init.cc:377] Logged in from keytab as oryx/127.28.177.190@KRBTEST.COM (short username oryx)
I20260808 02:17:55.205811 5560 webserver.cc:460] Webserver started at http://127.28.177.190:40775/ using document root <none> and password file <none>
I20260808 02:17:55.206103 5560 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:55.206156 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:55.206342 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:55.207697 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "50fe97b61719474d80bf4b8325421662"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "433ba6162d48b856026557e81d23c90f"
server_key_iv: "47d8b13b05a137ffe84faa2d693823de"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.208127 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "50fe97b61719474d80bf4b8325421662"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "433ba6162d48b856026557e81d23c90f"
server_key_iv: "47d8b13b05a137ffe84faa2d693823de"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.209836 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260808 02:17:55.210901 5577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.211199 5560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:55.211315 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "50fe97b61719474d80bf4b8325421662"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "433ba6162d48b856026557e81d23c90f"
server_key_iv: "47d8b13b05a137ffe84faa2d693823de"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.211397 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:55.221588 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:55.224049 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:55.224227 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:55.228826 5560 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42235
I20260808 02:17:55.228920 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42235 every 8 connection(s)
I20260808 02:17:55.229310 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:17:55.230410 5645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:55.232753 5645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Bootstrap starting.
I20260808 02:17:55.233448 5645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:55.233768 5645 log.cc:824] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:55.234608 5645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: No bootstrap required, opened a new log
I20260808 02:17:55.236436 5645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } }
I20260808 02:17:55.236588 5645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:55.236611 5645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fe97b61719474d80bf4b8325421662, State: Initialized, Role: FOLLOWER
I20260808 02:17:55.236789 5645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } }
I20260808 02:17:55.236904 5645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:17:55.236970 5645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:17:55.237025 5645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:55.237690 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 5560
I20260808 02:17:55.237846 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:17:55.237995 5645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } }
I20260808 02:17:55.238098 5645 leader_election.cc:302] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50fe97b61719474d80bf4b8325421662; no voters:
I20260808 02:17:55.238188 29382 external_mini_cluster.cc:1428] Setting key 69118c3c0762927c284f7dc23709e325
I20260808 02:17:55.238288 5645 leader_election.cc:288] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:17:55.238336 5649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:55.238555 5645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:55.238586 5649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 LEADER]: Becoming Leader. State: Replica: 50fe97b61719474d80bf4b8325421662, State: Running, Role: LEADER
I20260808 02:17:55.238698 5649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } }
I20260808 02:17:55.239246 5651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50fe97b61719474d80bf4b8325421662. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } }
I20260808 02:17:55.239351 5651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:55.239526 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } }
I20260808 02:17:55.239675 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:55.239599 5667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:55.241204 5667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:55.244957 5667 catalog_manager.cc:1347] Generated new cluster ID: 5acafbffa24548d7bde6b3b6009abd63
I20260808 02:17:55.245026 5667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:17:55.250582 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.238975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.28.177.190:42235)
0808 02:17:55.239132 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:55.239143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:55.239861 (+ 718us) server_negotiation.cc:407] Connection header received
0808 02:17:55.240017 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:55.240040 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:55.240128 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:55.240309 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:55.241097 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.242399 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.243311 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.243775 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.243921 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.246839 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:55.246843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:55.246843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:55.246858 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:55.248773 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.249214 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.249218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.249218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.249334 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.249583 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.249584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.249584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.249857 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:55.250094 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:55.250170 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:17:55.250332 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260808 02:17:55.271397 5667 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:17:55.272190 5667 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:55.284260 5667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Generated new TSK 0
I20260808 02:17:55.284493 5667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab.
Entry for principal oryx/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab.
I20260808 02:17:55.328577 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42235
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:55.417452 5682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:55.417721 5682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:55.417750 5682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:55.419111 5682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:55.419162 5682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:55.419193 5682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:55.420931 5682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:55.421142 5682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:55.421424 5682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:55.427479 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.427459 5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.427450 5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:55.427822 5682 server_base.cc:1034] running on GCE node
I20260808 02:17:55.428157 5682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:55.428395 5682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:55.429534 5682 hybrid_clock.cc:648] HybridClock initialized: now 1786155475429525 us; error 32 us; skew 500 ppm
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:55.432294 5682 init.cc:377] Logged in from keytab as oryx/127.28.177.129@KRBTEST.COM (short username oryx)
I20260808 02:17:55.432744 5682 webserver.cc:460] Webserver started at http://127.28.177.129:33529/ using document root <none> and password file <none>
I20260808 02:17:55.433040 5682 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:55.433120 5682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:55.433279 5682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:55.434459 5682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "63d8dcb828ef45e193672f5503fbf91e"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "aaabe2feca8c6436ce5dcf3b7c34d22c"
server_key_iv: "da647cb096a5eb4c18895003c5bf2e11"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.434828 5682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "63d8dcb828ef45e193672f5503fbf91e"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "aaabe2feca8c6436ce5dcf3b7c34d22c"
server_key_iv: "da647cb096a5eb4c18895003c5bf2e11"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.436172 5682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:17:55.436965 5699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.437203 5682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:55.437305 5682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "63d8dcb828ef45e193672f5503fbf91e"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "aaabe2feca8c6436ce5dcf3b7c34d22c"
server_key_iv: "da647cb096a5eb4c18895003c5bf2e11"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.437394 5682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:55.453289 5682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:55.455425 5682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:55.455608 5682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:55.455874 5682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:55.456233 5682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:55.456291 5682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.456336 5682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:55.456377 5682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.462487 5682 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:42297
I20260808 02:17:55.462600 5851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:42297 every 8 connection(s)
I20260808 02:17:55.462931 5682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:17:55.464725 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 5682
I20260808 02:17:55.464883 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:17:55.465163 29382 external_mini_cluster.cc:1428] Setting key 8081c8d4e0a64e1ce477e511561ef806
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:17:55.474555 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.463826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:40473 (local address 127.28.177.190:42235)
0808 02:17:55.463907 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:55.463909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:55.464056 (+ 147us) server_negotiation.cc:407] Connection header received
0808 02:17:55.464604 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:55.464606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:55.464621 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:55.464722 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:55.466092 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.466634 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.467745 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.468044 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.468112 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.470938 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:55.470940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:55.470941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:55.470952 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:55.472747 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.473292 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.473294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.473294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.473372 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.473708 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.473710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.473710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.473885 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:55.473961 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:55.474350 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 02:17:55.474447 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:55.474593 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.463749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:40473)
0808 02:17:55.463916 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:55.463960 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:55.464488 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:55.464890 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:55.464896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:55.465714 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:55.465963 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:55.465969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.466765 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.466766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:55.467607 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:55.467637 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.468149 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.468150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:55.468311 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.468939 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:55.468952 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:55.470830 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:55.472923 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:55.472938 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:55.472941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:55.473190 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:55.473473 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:55.473474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:55.473475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:55.473597 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:55.473966 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:55.474038 (+ 72us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:55.474182 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 02:17:55.474270 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:17:55.474407 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260808 02:17:55.475507 5852 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42235
I20260808 02:17:55.475626 5852 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:55.475822 5852 heartbeater.cc:511] Master 127.28.177.190:42235 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:17:55.476508 5596 ts_manager.cc:194] Registered new tserver with Master: 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129:42297)
I20260808 02:17:55.477224 5596 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:40473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab.
Entry for principal oryx/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab.
I20260808 02:17:55.496701 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42235
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:55.592026 5860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:55.592290 5860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:55.592334 5860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:55.593538 5860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:55.593611 5860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:55.593657 5860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:55.595294 5860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:55.595603 5860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:55.595868 5860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:55.601224 5871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.601191 5868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.601228 5869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:55.601526 5860 server_base.cc:1034] running on GCE node
I20260808 02:17:55.601804 5860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:55.602073 5860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:55.603289 5860 hybrid_clock.cc:648] HybridClock initialized: now 1786155475603266 us; error 40 us; skew 500 ppm
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:55.606454 5860 init.cc:377] Logged in from keytab as oryx/127.28.177.130@KRBTEST.COM (short username oryx)
I20260808 02:17:55.606994 5860 webserver.cc:460] Webserver started at http://127.28.177.130:43713/ using document root <none> and password file <none>
I20260808 02:17:55.607270 5860 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:55.607342 5860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:55.607496 5860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:55.608593 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "de1cb9d1f041494090fb3653a8bdb275"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "fcce6a72b71cf84aa0b863e90cab9620"
server_key_iv: "7c3e839d5d73fd540a3f30baa0569e73"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.608999 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "de1cb9d1f041494090fb3653a8bdb275"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "fcce6a72b71cf84aa0b863e90cab9620"
server_key_iv: "7c3e839d5d73fd540a3f30baa0569e73"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.610489 5860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:55.611358 5879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.611629 5860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:55.611752 5860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "de1cb9d1f041494090fb3653a8bdb275"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "fcce6a72b71cf84aa0b863e90cab9620"
server_key_iv: "7c3e839d5d73fd540a3f30baa0569e73"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.611838 5860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:55.624109 5860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:55.626717 5860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:55.626929 5860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:55.627206 5860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:55.627553 5860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:55.627612 5860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.627657 5860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:55.627684 5860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.634336 5860 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:44521
I20260808 02:17:55.634742 6038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:44521 every 8 connection(s)
I20260808 02:17:55.634975 5860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:17:55.643319 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 5860
I20260808 02:17:55.643424 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:17:55.643661 29382 external_mini_cluster.cc:1428] Setting key d6e440589d36d2608a9249c32681bc0a
I20260808 02:17:55.645780 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.635925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:57789 (local address 127.28.177.190:42235)
0808 02:17:55.636006 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:55.636008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:55.636289 (+ 281us) server_negotiation.cc:407] Connection header received
0808 02:17:55.636797 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:55.636799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:55.636810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:55.636917 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:55.637988 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.638630 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.639799 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.639988 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.640047 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.642558 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:55.642560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:55.642560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:55.642569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:55.644125 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.644640 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.644643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.644643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.644715 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.645015 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.645016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.645016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.645182 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:55.645261 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:55.645536 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 02:17:55.645637 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:55.645802 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.635901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.130:57789)
0808 02:17:55.636125 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:55.636174 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:55.636701 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:55.637032 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:55.637036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:55.637650 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:55.637857 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:55.637862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.638731 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.638732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:55.639727 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:55.639730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.640072 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.640073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:55.640186 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.640902 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:55.640912 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:55.642444 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:55.644245 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:55.644252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:55.644254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:55.644564 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:55.644789 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:55.644790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:55.644790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:55.644948 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:55.645262 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:55.645268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:55.645399 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 02:17:55.645492 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:17:55.645628 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260808 02:17:55.646399 6039 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42235
I20260808 02:17:55.646535 6039 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:55.646732 6039 heartbeater.cc:511] Master 127.28.177.190:42235 requested a full tablet report, sending...
I20260808 02:17:55.647215 5596 ts_manager.cc:194] Registered new tserver with Master: de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521)
I20260808 02:17:55.647665 5596 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:57789
WARNING: no policy specified for oryx/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab.
Entry for principal oryx/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab.
I20260808 02:17:55.674294 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42235
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:55.762420 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:55.762720 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:55.762770 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:55.764191 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:55.764288 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:55.764343 6046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:55.766188 6046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:55.766496 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:55.766768 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:55.772403 6060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.772457 6055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.772435 6056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:55.772567 6046 server_base.cc:1034] running on GCE node
I20260808 02:17:55.773088 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:55.773367 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:55.774533 6046 hybrid_clock.cc:648] HybridClock initialized: now 1786155475774515 us; error 28 us; skew 500 ppm
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:55.777390 6046 init.cc:377] Logged in from keytab as oryx/127.28.177.131@KRBTEST.COM (short username oryx)
I20260808 02:17:55.777868 6046 webserver.cc:460] Webserver started at http://127.28.177.131:43411/ using document root <none> and password file <none>
I20260808 02:17:55.778124 6046 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:55.778177 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:55.778342 6046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:55.779453 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "8fe525ab07cd4df1a2738e4f62f37db4"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "717e5229dc4561282df022b136e5dceb"
server_key_iv: "61d60db596db681280c636817f50d9e3"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.779838 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "8fe525ab07cd4df1a2738e4f62f37db4"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "717e5229dc4561282df022b136e5dceb"
server_key_iv: "61d60db596db681280c636817f50d9e3"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.781392 6046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:55.782330 6067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.782541 6046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:55.782672 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "8fe525ab07cd4df1a2738e4f62f37db4"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "717e5229dc4561282df022b136e5dceb"
server_key_iv: "61d60db596db681280c636817f50d9e3"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.782735 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:55.810783 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:55.813941 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:55.814083 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:55.814301 6046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:55.814684 6046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:55.814754 6046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.814790 6046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:55.814821 6046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.821552 6046 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:43219
I20260808 02:17:55.821606 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:43219 every 8 connection(s)
I20260808 02:17:55.821971 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:17:55.831290 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 6046
I20260808 02:17:55.831415 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:17:55.831687 29382 external_mini_cluster.cc:1428] Setting key 5b547803f66f4b0207da089b1ccff6c1
I20260808 02:17:55.832490 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.822913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:35053 (local address 127.28.177.190:42235)
0808 02:17:55.822987 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:55.822989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:55.823108 (+ 119us) server_negotiation.cc:407] Connection header received
0808 02:17:55.823619 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:55.823620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:55.823632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:55.823703 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:55.824879 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.825446 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.826482 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.826670 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.826728 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.829353 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:55.829354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:55.829355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:55.829363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:55.830661 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.831128 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.831129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.831130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.831234 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:55.831586 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:55.831587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:55.831587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:55.831748 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:55.831832 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:55.832260 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:17:55.832360 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:17:55.832782 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:55.822777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.131:35053)
0808 02:17:55.822957 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:55.823004 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:55.823527 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:55.823771 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:55.823775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:55.824529 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:55.824736 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:55.824741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.825560 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.825562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:55.826387 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:55.826389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:55.826760 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:55.826761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:55.826878 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:55.827539 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:55.827549 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:55.829249 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:55.830762 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:55.830769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:55.830771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:55.831007 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:55.831314 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:55.831321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:55.831322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:55.831486 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:55.831852 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:55.831860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:55.832064 (+ 204us) client_negotiation.cc:769] Sending connection context
0808 02:17:55.832219 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 02:17:55.832482 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260808 02:17:55.833424 6233 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42235
I20260808 02:17:55.833552 6233 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:55.833771 6233 heartbeater.cc:511] Master 127.28.177.190:42235 requested a full tablet report, sending...
I20260808 02:17:55.834187 5596 ts_manager.cc:194] Registered new tserver with Master: 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:55.834642 5596 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:35053
I20260808 02:17:55.844431 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.177.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.189.keytab.
Entry for principal oryx/127.28.177.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.189.keytab.
I20260808 02:17:55.877228 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:38019
--webserver_interface=127.28.177.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.189.keytab
--principal=oryx/127.28.177.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42235,127.28.177.189:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:55.967752 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:55.967983 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:55.968005 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:55.969410 6240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:55.969466 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:55.969482 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:55.969530 6240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:55.969553 6240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:55.971095 6240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42235,127.28.177.189:38019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.189.keytab
--principal=oryx/127.28.177.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.177.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
W20260808 02:17:55.971246 6240 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.177.190:42235,127.28.177.189:38019'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 02:17:55.971407 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:55.971650 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:55.977237 6248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.977223 6250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:55.977221 6247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:55.977662 6240 server_base.cc:1034] running on GCE node
I20260808 02:17:55.977911 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:55.978257 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:55.979413 6240 hybrid_clock.cc:648] HybridClock initialized: now 1786155475979399 us; error 32 us; skew 500 ppm
Aug 08 02:17:55 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:55.982264 6240 init.cc:377] Logged in from keytab as oryx/127.28.177.189@KRBTEST.COM (short username oryx)
I20260808 02:17:55.982805 6240 webserver.cc:460] Webserver started at http://127.28.177.189:41919/ using document root <none> and password file <none>
I20260808 02:17:55.983075 6240 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:55.983158 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:55.983312 6240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:55.984431 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/instance:
uuid: "f6970c77282b4a55aaf31075a30ba57f"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "04e2551c630449da3ddc81d307c65e30"
server_key_iv: "33c6469dd7a8483ca15f24ae480ca81e"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.984808 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal/instance:
uuid: "f6970c77282b4a55aaf31075a30ba57f"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "04e2551c630449da3ddc81d307c65e30"
server_key_iv: "33c6469dd7a8483ca15f24ae480ca81e"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.986326 6240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:55.987154 6259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:55.987376 6240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:55.987490 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
uuid: "f6970c77282b4a55aaf31075a30ba57f"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "04e2551c630449da3ddc81d307c65e30"
server_key_iv: "33c6469dd7a8483ca15f24ae480ca81e"
server_key_version: "encryptionkey@0"
I20260808 02:17:55.987583 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:55.995395 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:55.997380 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:55.997550 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.001840 6240 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.189:38019
I20260808 02:17:56.001873 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.189:38019 every 8 connection(s)
I20260808 02:17:56.002233 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
I20260808 02:17:56.003340 6332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:56.004005 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 6240
I20260808 02:17:56.005477 6332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.189@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
I20260808 02:17:56.016355 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.006262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.28.177.190:42235)
0808 02:17:56.006319 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.006321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.006442 (+ 121us) server_negotiation.cc:407] Connection header received
0808 02:17:56.006496 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.006497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.006508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.006573 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.007053 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.007802 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.009145 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.009435 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.009501 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.012014 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.012016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.012016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.012025 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.014308 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.014967 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.015085 (+ 118us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.015085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.015183 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.015516 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.015517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.015517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.015816 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.015903 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.016158 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.016259 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:17:56.016556 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.006180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:34654)
0808 02:17:56.006344 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.006382 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.006450 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.006584 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.006586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.006827 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.006945 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.006950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.007912 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.007913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.009061 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.009064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.009529 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.009530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.009608 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.010239 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.010249 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.011915 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.014499 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.014505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.014508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.014860 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.015272 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.015274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.015274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.015428 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.015891 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.015898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.016013 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.016122 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.016302 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":18,"thread_start_us":106,"threads_started":1}
I20260808 02:17:56.016571 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.004857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.28.177.189:38019)
0808 02:17:56.005041 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.005050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.005085 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:17:56.005482 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.005489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.005561 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.005678 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.006432 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.007649 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.008759 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.009299 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.009415 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.012572 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.012586 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.012586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.012596 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.014986 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.015432 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.015435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.015436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.015545 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.015776 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.015777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.015777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.016081 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.016256 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.016349 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.016441 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260808 02:17:56.017449 6332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.189@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
I20260808 02:17:56.024304 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.017730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48542)
0808 02:17:56.017781 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.017792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.017833 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.017950 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.017951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.018061 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.018108 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.018109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.018752 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.018752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.019390 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.019393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.019757 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.019758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.019822 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.020260 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.020264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.021632 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.023237 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.023239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.023239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.023419 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.023747 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.023748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.023748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.023810 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.024057 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.024058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.024093 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.024148 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.024235 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:17:56.024374 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.017781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.28.177.189:38019)
0808 02:17:56.017831 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.017833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.017842 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:56.017890 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.017890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.017898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.017946 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.018195 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.018686 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.019487 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.019672 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.019729 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.021726 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.021727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.021727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.021734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.022987 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.023550 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.023551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.023551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.023621 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.023881 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.023882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.023883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.023995 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.024054 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.024166 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.024286 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":32}
I20260808 02:17:56.025959 6332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Bootstrap starting.
I20260808 02:17:56.026911 6332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:56.027163 6332 log.cc:824] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:56.028008 6332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: No bootstrap required, opened a new log
I20260808 02:17:56.029660 6332 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } }
I20260808 02:17:56.029798 6332 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:56.029824 6332 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6970c77282b4a55aaf31075a30ba57f, State: Initialized, Role: FOLLOWER
I20260808 02:17:56.029927 6332 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } }
I20260808 02:17:56.030202 6343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } } }
I20260808 02:17:56.030270 6332 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:56.030290 6343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:17:56.032100 6360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:17:56.032188 6360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:17:56.034287 6240 master_runner.cc:428] Detected that this master f6970c77282b4a55aaf31075a30ba57f is joining an existing cluster
I20260808 02:17:56.034360 6240 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.177.189:38019
I20260808 02:17:56.043783 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.035678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.28.177.190:42235)
0808 02:17:56.035835 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.035838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.035958 (+ 120us) server_negotiation.cc:407] Connection header received
0808 02:17:56.036083 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.036085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.036102 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.036157 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.036736 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.037638 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.038618 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.038864 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.038930 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.040310 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.040312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.040312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.040320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.042098 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.042665 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.042667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.042667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.042727 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.042909 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.042910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.042910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.043119 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.043192 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.043595 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.043681 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260808 02:17:56.043793 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.035575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:34670)
0808 02:17:56.035890 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.035899 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.035972 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.036208 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.036211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.036541 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.036642 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.036644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.037736 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.037737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.038535 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.038537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.038997 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.038997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.039056 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.039740 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.039744 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.040215 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.042183 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.042184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.042185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.042569 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.042797 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.042797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.042797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.042845 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.043217 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.043218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.043262 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.043344 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.043451 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260808 02:17:56.043905 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.035762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.28.177.189:38019)
0808 02:17:56.035825 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.035828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.036166 (+ 338us) server_negotiation.cc:407] Connection header received
0808 02:17:56.036194 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.036195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.036206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.036284 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.036914 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.037391 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.038438 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.038773 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.038857 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.040295 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.040297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.040297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.040305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.042603 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.043048 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.043049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.043050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.043100 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.043340 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.043341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.043341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.043462 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.043536 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.043728 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.043812 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:17:56.044200 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.035576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48550)
0808 02:17:56.036076 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.036086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.036141 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.036302 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.036303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.036558 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.036642 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.036644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.037488 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.037489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.038348 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.038350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.038876 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.038877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.038941 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.039561 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.039567 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.040172 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.042689 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.042690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.042691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.042923 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.043185 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.043186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.043187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.043261 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.043550 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.043551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.043598 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.043978 (+ 380us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.044084 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":3,"spinlock_wait_cycles":9600,"thread_start_us":124,"threads_started":1}
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.190@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
I20260808 02:17:56.054263 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.046779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.28.177.189:38019)
0808 02:17:56.046818 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.046820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.046984 (+ 164us) server_negotiation.cc:407] Connection header received
0808 02:17:56.047078 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.047079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.047090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.047144 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.047615 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.048139 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.048985 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.049167 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.049241 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.051482 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.051484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.051484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.051492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.053018 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.053435 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.053437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.053437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.053499 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.053714 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.053715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.053715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.053829 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.053904 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.054097 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.054171 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:17:56.054287 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.046685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48564)
0808 02:17:56.046864 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.046913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.047009 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.047155 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.047159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.047442 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.047521 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.047523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.048232 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.048234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.048888 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.048891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.049253 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.049255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.049336 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.049802 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.049812 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.051387 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.053120 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.053129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.053131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.053342 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.053567 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.053569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.053569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.053638 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.053897 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.053905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.053958 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.054058 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.054177 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260808 02:17:56.055177 5596 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.177.189:38019: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f"
member_type: NON_VOTER
last_known_addr {
host: "127.28.177.189"
port: 38019
}
attrs {
promote: true
}
}
dest_uuid: "50fe97b61719474d80bf4b8325421662"
cas_config_opid_index: -1
I20260808 02:17:56.055575 5596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } }
I20260808 02:17:56.056074 6373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f6970c77282b4a55aaf31075a30ba57f (127.28.177.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } }
I20260808 02:17:56.056421 6292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:56.057140 6373 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.177.189:38019
I20260808 02:17:56.057403 6372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } }
I20260808 02:17:56.057514 6292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Refusing update from remote peer 50fe97b61719474d80bf4b8325421662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 02:17:56.057615 6240 master.cc:537] Master@127.28.177.189:38019 shutting down...
I20260808 02:17:56.057608 6372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.058350 6372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:17:56.059096 5578 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 02:17:56.059952 5578 consensus_peers.cc:487] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 -> Peer f6970c77282b4a55aaf31075a30ba57f (127.28.177.189:38019): Couldn't send request to peer f6970c77282b4a55aaf31075a30ba57f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 02:17:56.060897 6240 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:56.061017 6240 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:56.061072 6240 tablet_replica.cc:330] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: stopping tablet replica
I20260808 02:17:56.074115 6240 master.cc:552] Master@127.28.177.189:38019 shutdown complete.
I20260808 02:17:56.074194 6240 master_runner.cc:305] Clearing existing system tablet
I20260808 02:17:56.074348 6240 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.074404 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.075410 6240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:56.076021 6379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.076169 6240 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.076238 6240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.076258 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
uuid: "f6970c77282b4a55aaf31075a30ba57f"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "04e2551c630449da3ddc81d307c65e30"
server_key_iv: "33c6469dd7a8483ca15f24ae480ca81e"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.077148 6240 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:56.078661 6240 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:17:56.078733 6240 log.cc:1196] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 02:17:56.078962 6240 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Deleting consensus metadata
I20260808 02:17:56.079116 6240 master_runner.cc:315] Copying system tablet from 127.28.177.190:42235
I20260808 02:17:56.079769 6240 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: tablet copy: Beginning tablet copy session from remote peer at address 127.28.177.190:42235
I20260808 02:17:56.087392 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.080287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.28.177.190:42235)
0808 02:17:56.080371 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.080374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.080540 (+ 166us) server_negotiation.cc:407] Connection header received
0808 02:17:56.080646 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.080647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.080663 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.080736 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.081229 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.081809 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.082753 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.082993 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.083097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.084219 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.084221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.084221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.084230 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.085770 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.086412 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.086414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.086414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.086487 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.086744 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.086745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.086746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.086876 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.086959 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.087214 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.087295 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:17:56.087409 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.080126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:34676)
0808 02:17:56.080404 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.080418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.080561 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.080742 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.080743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.081060 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.081114 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.081117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.081912 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.081914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.082618 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.082621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.083098 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.083099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.083180 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.083762 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.083768 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.084099 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.085858 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.085859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.085859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.086078 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.086585 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.086586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.086587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.086664 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.086947 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.086948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.086997 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.087172 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.087295 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260808 02:17:56.087802 5626 tablet_copy_service.cc:140] P 50fe97b61719474d80bf4b8325421662: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f6970c77282b4a55aaf31075a30ba57f ({username='oryx', principal='oryx/127.28.177.189@KRBTEST.COM'} at 127.0.0.1:34676)
I20260808 02:17:56.087942 5626 tablet_copy_service.cc:161] P 50fe97b61719474d80bf4b8325421662: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f6970c77282b4a55aaf31075a30ba57f at {username='oryx', principal='oryx/127.28.177.189@KRBTEST.COM'} at 127.0.0.1:34676: session id = f6970c77282b4a55aaf31075a30ba57f-00000000000000000000000000000000
I20260808 02:17:56.089080 5626 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:17:56.089879 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:17:56.090906 6240 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: tablet copy: Starting download of 0 data blocks...
I20260808 02:17:56.091086 6240 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: tablet copy: Starting download of 1 WAL segments...
I20260808 02:17:56.092173 6240 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:17:56.093183 5626 tablet_copy_service.cc:342] P 50fe97b61719474d80bf4b8325421662: Request end of tablet copy session f6970c77282b4a55aaf31075a30ba57f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.177.189@KRBTEST.COM'} at 127.0.0.1:34676
I20260808 02:17:56.093250 5626 tablet_copy_service.cc:433] P 50fe97b61719474d80bf4b8325421662: ending tablet copy session f6970c77282b4a55aaf31075a30ba57f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f6970c77282b4a55aaf31075a30ba57f
I20260808 02:17:56.096243 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:56.097707 6388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.097726 6389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.097764 6240 server_base.cc:1034] running on GCE node
W20260808 02:17:56.097926 6392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.098238 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:56.098400 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:56.099545 6240 hybrid_clock.cc:648] HybridClock initialized: now 1786155476099535 us; error 22 us; skew 500 ppm
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:56.101329 6240 init.cc:377] Logged in from keytab as oryx/127.28.177.189@KRBTEST.COM (short username oryx)
I20260808 02:17:56.101678 6240 webserver.cc:460] Webserver started at http://127.28.177.189:38031/ using document root <none> and password file <none>
I20260808 02:17:56.101814 6240 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.101855 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.102694 6240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:56.103386 6402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.103561 6240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.103621 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
uuid: "f6970c77282b4a55aaf31075a30ba57f"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "04e2551c630449da3ddc81d307c65e30"
server_key_iv: "33c6469dd7a8483ca15f24ae480ca81e"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.103683 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.132084 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:56.134435 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.137880 6240 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.189:38019
I20260808 02:17:56.138123 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.189:38019 every 8 connection(s)
I20260808 02:17:56.138361 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
I20260808 02:17:56.139289 6476 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:17:56.139906 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Bootstrap starting.
I20260808 02:17:56.142393 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:17:56.142652 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: Bootstrap complete.
I20260808 02:17:56.142882 6476 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } }
I20260808 02:17:56.142954 6476 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6970c77282b4a55aaf31075a30ba57f, State: Initialized, Role: LEARNER
I20260808 02:17:56.142982 6476 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } }
I20260808 02:17:56.143169 6476 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:56.143196 6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } }
I20260808 02:17:56.143306 6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: LEARNER
I20260808 02:17:56.144683 6503 catalog_manager.cc:1259] Loaded cluster ID: 5acafbffa24548d7bde6b3b6009abd63
I20260808 02:17:56.144750 6503 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: loading cluster ID for follower catalog manager: success
I20260808 02:17:56.145681 6503 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: acquiring CA information for follower catalog manager: success
I20260808 02:17:56.145968 6503 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:17:56.177366 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.169268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.28.177.189:38019)
0808 02:17:56.169424 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.169426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.169435 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:56.169462 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.169462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.169472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.169579 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.170337 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.171367 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.172487 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.172801 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.172921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.174461 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.174463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.174463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.174471 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.176139 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.176481 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.176482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.176482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.176560 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.176762 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.176763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.176763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.176939 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.177052 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.177134 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.177215 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.28.177.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.188.keytab.
Entry for principal oryx/127.28.177.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.188.keytab.
I20260808 02:17:56.208446 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:33859
--webserver_interface=127.28.177.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.188.keytab
--principal=oryx/127.28.177.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:17:56.296188 6511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:56.296396 6511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:56.296417 6511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:56.297662 6511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:17:56.297703 6511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:56.297719 6511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:56.297730 6511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:17:56.297742 6511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:17:56.299235 6511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.188.keytab
--principal=oryx/127.28.177.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.177.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:56.299445 6511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:56.299629 6511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:56.304708 6517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.304694 6520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.305048 6518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.305091 6511 server_base.cc:1034] running on GCE node
I20260808 02:17:56.305392 6511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:56.305689 6511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:56.306854 6511 hybrid_clock.cc:648] HybridClock initialized: now 1786155476306837 us; error 33 us; skew 500 ppm
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:56.309800 6511 init.cc:377] Logged in from keytab as oryx/127.28.177.188@KRBTEST.COM (short username oryx)
I20260808 02:17:56.310369 6511 webserver.cc:460] Webserver started at http://127.28.177.188:38427/ using document root <none> and password file <none>
I20260808 02:17:56.310664 6511 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.310722 6511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.310900 6511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:17:56.312038 6511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/instance:
uuid: "b11f30b06888407d9ef5377a1b6cd8e0"
format_stamp: "Formatted at 2026-08-08 02:17:56 on dist-test-slave-3h5h"
server_key: "fe1c7d5b1955655488b9424acddf3d68"
server_key_iv: "5191c6734a4ac6a83c437db24790e024"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.312448 6511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal/instance:
uuid: "b11f30b06888407d9ef5377a1b6cd8e0"
format_stamp: "Formatted at 2026-08-08 02:17:56 on dist-test-slave-3h5h"
server_key: "fe1c7d5b1955655488b9424acddf3d68"
server_key_iv: "5191c6734a4ac6a83c437db24790e024"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.314085 6511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:17:56.315083 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.315333 6511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:56.315423 6511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
uuid: "b11f30b06888407d9ef5377a1b6cd8e0"
format_stamp: "Formatted at 2026-08-08 02:17:56 on dist-test-slave-3h5h"
server_key: "fe1c7d5b1955655488b9424acddf3d68"
server_key_iv: "5191c6734a4ac6a83c437db24790e024"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.315553 6511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.338707 6511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:56.341372 6511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:56.341583 6511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.346916 6511 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.188:33859
I20260808 02:17:56.346957 6601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.188:33859 every 8 connection(s)
I20260808 02:17:56.347380 6511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
I20260808 02:17:56.348459 6602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:56.350112 6602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:17:56.354894 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 6511
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.188@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.177.188@KRBTEST.COM
I20260808 02:17:56.360473 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.351191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.28.177.190:42235)
0808 02:17:56.351277 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.351279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.351478 (+ 199us) server_negotiation.cc:407] Connection header received
0808 02:17:56.351955 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.351956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.351968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.352035 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.352734 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.353345 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.354394 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.354577 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.354632 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.357333 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.357336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.357336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.357344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.358935 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.359402 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.359404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.359404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.359473 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.359736 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.359737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.359737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.359900 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.359984 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.360286 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.360371 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:17:56.360590 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.350915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:34692)
0808 02:17:56.351321 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.351366 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.351853 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.352114 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.352118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.352386 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.352630 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.352635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.353478 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.353479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.354309 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.354311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.354725 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.354727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.354918 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.355558 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.355569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.357213 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.359053 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.359063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.359066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.359316 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.359542 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.359543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.359544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.359660 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.359996 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.360002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.360127 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.360232 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.360363 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":119,"threads_started":1}
I20260808 02:17:56.361361 6602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:17:56.364336 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.355600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.28.177.188:33859)
0808 02:17:56.355786 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.355790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.355808 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:17:56.355826 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.355827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.355843 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.355926 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.356538 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.357344 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.358079 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.358396 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.358483 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.361027 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.361035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.361035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.361042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.363062 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.363441 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.363444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.363445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.363510 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.363698 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.363699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.363699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.363876 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.364015 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.364127 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.364229 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.188@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
I20260808 02:17:56.368485 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.361679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48596)
0808 02:17:56.361729 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.361741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.361782 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.361987 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.361988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.362153 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.362222 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.362223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.362980 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.362981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.363758 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.363760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.364231 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.364232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.364306 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.364704 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.364709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.366023 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.367536 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.367537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.367537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.367728 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.367943 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.367943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.367944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.368007 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.368243 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.368243 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.368278 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.368331 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.368422 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:17:56.368492 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.361799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.28.177.189:38019)
0808 02:17:56.361849 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.361851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.361860 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:56.361889 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.361889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.361910 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.361966 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.362329 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.362887 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.363839 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.364152 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.364219 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.366117 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.366119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.366119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.366127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.367441 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.367802 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.367803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.367803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.367857 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.368076 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.368077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.368077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.368179 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.368237 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.368347 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.368417 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:17:56.369057 6602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.188@KRBTEST.COM for oryx/127.28.177.188@KRBTEST.COM
I20260808 02:17:56.375641 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.369324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55568)
0808 02:17:56.369375 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.369385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.369421 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.369525 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.369525 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.369655 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.369682 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.369683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.370314 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.370315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.370788 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.370790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.371180 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.371181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.371227 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.371590 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.371594 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.372712 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.374553 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.374554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.374555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.374710 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.374926 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.374927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.374928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.374987 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.375412 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.375413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.375448 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.375503 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.375580 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:17:56.375725 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.369398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.28.177.188:33859)
0808 02:17:56.369435 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.369437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.369444 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:17:56.369469 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.369469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.369479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.369524 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.369757 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.370223 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.370882 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.371096 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.371182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.372805 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.372806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.372806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.372813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.374467 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.374781 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.374782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.374782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.374839 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.375216 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.375217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.375218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.375339 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.375409 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.375544 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.375633 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":20}
I20260808 02:17:56.377379 6602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Bootstrap starting.
I20260808 02:17:56.378258 6602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:56.378556 6602 log.cc:824] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:56.379472 6602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: No bootstrap required, opened a new log
I20260808 02:17:56.381009 6602 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } }
I20260808 02:17:56.381165 6602 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:56.381214 6602 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11f30b06888407d9ef5377a1b6cd8e0, State: Initialized, Role: FOLLOWER
I20260808 02:17:56.381352 6602 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } }
I20260808 02:17:56.381633 6614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } } }
I20260808 02:17:56.381701 6602 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:56.381752 6614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:17:56.383591 6630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:17:56.383677 6630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:17:56.386579 6511 master_runner.cc:428] Detected that this master b11f30b06888407d9ef5377a1b6cd8e0 is joining an existing cluster
I20260808 02:17:56.386637 6511 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.177.188:33859
I20260808 02:17:56.395887 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.388162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55578)
0808 02:17:56.388533 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.388543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.388609 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.388775 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.388776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.389166 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.389226 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.389228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.390198 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.390200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.391248 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.391251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.391786 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.391849 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.392491 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.392496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.392966 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.394710 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.394711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.394712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.394910 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.395148 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.395149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.395149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.395212 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.395494 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.395495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.395541 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.395611 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.395736 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":64,"threads_started":1}
I20260808 02:17:56.395946 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.388426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.28.177.188:33859)
0808 02:17:56.388480 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.388482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.388587 (+ 105us) server_negotiation.cc:407] Connection header received
0808 02:17:56.388678 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.388678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.388688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.388717 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.389306 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.390081 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391353 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.391654 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391695 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.393153 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.393155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.393155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.393163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.394603 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.395005 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.395006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.395007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.395072 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.395292 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.395293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.395294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.395418 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.395504 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.395646 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.395817 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":22}
I20260808 02:17:56.416210 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 5682
I20260808 02:17:56.424129 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.387937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.28.177.189:38019)
0808 02:17:56.387997 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.388000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.388217 (+ 217us) server_negotiation.cc:407] Connection header received
0808 02:17:56.388346 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.388348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.388359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.388415 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.389207 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.390082 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391356 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.391877 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391927 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.420952 (+ 29025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.420954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.420955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.420967 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.422828 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423276 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.423277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.423277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.423340 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423538 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.423539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.423539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.423679 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.423723 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.423928 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.424021 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:17:56.424485 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.387984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48600)
0808 02:17:56.388136 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.388149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.388251 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.388655 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.388656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.389002 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.389093 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.389096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.390212 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.390213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.391250 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.391251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.392014 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.392015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.392090 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.419424 (+ 27334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.419434 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.420235 (+ 801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.422934 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.422935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.422936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.423184 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.423412 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.423413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.423472 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.423772 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.423773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.423821 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.424248 (+ 427us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.424372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":86,"thread_start_us":79,"threads_started":1}
I20260808 02:17:56.425751 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.388380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.28.177.190:42235)
0808 02:17:56.388431 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.388434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.388444 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:56.388509 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.388510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.388524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.388598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.389293 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.390064 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391314 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.391621 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391679 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.421353 (+ 29674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.421356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.421356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.421368 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.423197 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423698 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.423699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.423700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.423768 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423999 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.424000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.424001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.424172 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.424229 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.425479 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.425584 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 02:17:56.426231 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.387886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:34696)
0808 02:17:56.388134 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.388146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.388201 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.388614 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.388615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.389099 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.389182 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.389184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.390171 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.390172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.391204 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.391206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.391780 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.391781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.391846 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.419510 (+ 27664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.419516 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.420797 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.423320 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.423322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.423585 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.423850 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.423851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.423851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.423920 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.425284 (+ 1364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.425285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.425343 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.425903 (+ 560us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.426051 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":84,"thread_start_us":123,"threads_started":1}
I20260808 02:17:56.427429 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:42297
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:42235
--webserver_port=33529
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155475, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.190@KRBTEST.COM for oryx/127.28.177.188@KRBTEST.COM
I20260808 02:17:56.437955 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.428824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55590)
0808 02:17:56.428973 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.428986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.429144 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.429337 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.429339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.429528 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.429583 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.429586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.430624 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.430625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.431473 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.431476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.432058 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.432059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.432133 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.432714 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.432719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.434364 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.436567 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.436568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.436569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.436819 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.437107 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.437109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.437109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.437240 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.437604 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.437605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.437658 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.437723 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.437841 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260808 02:17:56.438400 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.428901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.28.177.188:33859)
0808 02:17:56.428948 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.428951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.429117 (+ 166us) server_negotiation.cc:407] Connection header received
0808 02:17:56.429222 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.429223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.429237 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.429304 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.429680 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.430514 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.431578 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.431972 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.432061 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.434476 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.434479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.434479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.434490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.436459 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.436927 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.436929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.436930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.437007 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.437336 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.437338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.437338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.437520 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.437603 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.437800 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.437898 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:56.439059 5596 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.177.188:33859: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0"
member_type: NON_VOTER
last_known_addr {
host: "127.28.177.188"
port: 33859
}
attrs {
promote: true
}
}
dest_uuid: "50fe97b61719474d80bf4b8325421662"
cas_config_opid_index: 5
I20260808 02:17:56.439364 5596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } }
I20260808 02:17:56.439793 6373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.440419 6567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:56.441120 6373 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.177.188:33859
I20260808 02:17:56.441215 6373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.441285 6373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.441605 6511 master.cc:537] Master@127.28.177.188:33859 shutting down...
I20260808 02:17:56.441908 6567 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Refusing update from remote peer 50fe97b61719474d80bf4b8325421662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:17:56.442224 6372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:17:56.442919 5579 consensus_peers.cc:487] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 -> Peer b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188:33859): Couldn't send request to peer b11f30b06888407d9ef5377a1b6cd8e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 02:17:56.444129 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.439887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48616)
0808 02:17:56.439954 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.439968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.440016 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.440439 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.440440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.440441 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:56.440483 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.440485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.441586 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.441587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.442946 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.442948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.443841 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.443843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.443913 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.443930 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.444012 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.444055 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 02:17:56.444137 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.440026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.28.177.189:38019)
0808 02:17:56.440232 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.440234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.440244 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:56.440271 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.440272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.440297 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.440371 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:56.440561 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.441483 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.443049 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.443637 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.443719 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.443800 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.444019 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.444060 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260808 02:17:56.444401 6442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 LEARNER]: Refusing update from remote peer 50fe97b61719474d80bf4b8325421662: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:17:56.444646 6372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 02:17:56.445751 6442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.446045 6511 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:56.446162 6511 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:56.446238 6511 tablet_replica.cc:330] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: stopping tablet replica
I20260808 02:17:56.447302 6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50fe97b61719474d80bf4b8325421662. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.447423 6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: LEARNER
I20260808 02:17:56.447774 6372 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: attempting to promote NON_VOTER f6970c77282b4a55aaf31075a30ba57f to VOTER
I20260808 02:17:56.448032 6372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } }
I20260808 02:17:56.448757 6442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 LEARNER]: Refusing update from remote peer 50fe97b61719474d80bf4b8325421662: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:17:56.448998 6373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 02:17:56.449304 6650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: NON_VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: true } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.449404 6650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: LEARNER
I20260808 02:17:56.449817 6373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f6970c77282b4a55aaf31075a30ba57f (127.28.177.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.450076 6511 master.cc:552] Master@127.28.177.188:33859 shutdown complete.
I20260808 02:17:56.450116 6511 master_runner.cc:305] Clearing existing system tablet
I20260808 02:17:56.450263 6511 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.450315 6511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.450748 6373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.450824 6373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.450868 6373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.450915 6373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.451126 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.448589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55596)
0808 02:17:56.451034 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.451091 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2423}
I20260808 02:17:56.451218 6437 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f6970c77282b4a55aaf31075a30ba57f (127.28.177.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.451529 6511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:56.452267 6650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.452422 6650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:17:56.452648 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.452429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55610)
0808 02:17:56.452586 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.452617 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260808 02:17:56.452726 5579 consensus_peers.cc:487] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 -> Peer b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188:33859): Couldn't send request to peer b11f30b06888407d9ef5377a1b6cd8e0. Status: Network error: Client connection negotiation failed: client connection to 127.28.177.188:33859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:17:56.452896 6655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.453034 6511 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.453099 6511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:56.453126 6511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
uuid: "b11f30b06888407d9ef5377a1b6cd8e0"
format_stamp: "Formatted at 2026-08-08 02:17:56 on dist-test-slave-3h5h"
server_key: "fe1c7d5b1955655488b9424acddf3d68"
server_key_iv: "5191c6734a4ac6a83c437db24790e024"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.453961 6511 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:56.455441 6511 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:17:56.455509 6511 log.cc:1196] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 02:17:56.455719 6511 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Deleting consensus metadata
I20260808 02:17:56.455955 6511 master_runner.cc:315] Copying system tablet from 127.28.177.190:42235
I20260808 02:17:56.456699 6511 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.177.190:42235
I20260808 02:17:56.468510 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.457145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.28.177.190:42235)
0808 02:17:56.457199 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.457201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.461117 (+ 3916us) server_negotiation.cc:407] Connection header received
0808 02:17:56.461213 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.461215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.461229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.461290 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.461732 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.462536 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.463728 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.464018 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.464101 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.465365 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.465367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.465367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.465376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.467132 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.467605 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.467607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.467607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.467681 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.467925 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.467926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.467926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.468076 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.468213 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.468325 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.468415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:17:56.468539 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.457048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:34712)
0808 02:17:56.461003 (+ 3955us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.461019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.461124 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.461317 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.461318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.461555 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.461624 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.461627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.462647 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.462649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.463628 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.463632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.464124 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.464125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.464187 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.464775 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.464782 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.465263 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.467244 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.467245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.467246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.467510 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.467766 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.467768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.467769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.467847 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.468158 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.468159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.468214 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.468305 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.468429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3926,"thread_start_us":77,"threads_started":1}
I20260808 02:17:56.469381 5626 tablet_copy_service.cc:140] P 50fe97b61719474d80bf4b8325421662: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b11f30b06888407d9ef5377a1b6cd8e0 ({username='oryx', principal='oryx/127.28.177.188@KRBTEST.COM'} at 127.0.0.1:34712)
I20260808 02:17:56.469451 5626 tablet_copy_service.cc:161] P 50fe97b61719474d80bf4b8325421662: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b11f30b06888407d9ef5377a1b6cd8e0 at {username='oryx', principal='oryx/127.28.177.188@KRBTEST.COM'} at 127.0.0.1:34712: session id = b11f30b06888407d9ef5377a1b6cd8e0-00000000000000000000000000000000
I20260808 02:17:56.470098 5626 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:17:56.470709 6511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:17:56.472173 6511 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: tablet copy: Starting download of 0 data blocks...
I20260808 02:17:56.472328 6511 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: tablet copy: Starting download of 1 WAL segments...
I20260808 02:17:56.473297 6511 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:17:56.474438 5626 tablet_copy_service.cc:342] P 50fe97b61719474d80bf4b8325421662: Request end of tablet copy session b11f30b06888407d9ef5377a1b6cd8e0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.177.188@KRBTEST.COM'} at 127.0.0.1:34712
I20260808 02:17:56.474498 5626 tablet_copy_service.cc:433] P 50fe97b61719474d80bf4b8325421662: ending tablet copy session b11f30b06888407d9ef5377a1b6cd8e0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b11f30b06888407d9ef5377a1b6cd8e0
I20260808 02:17:56.477234 6511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:56.478747 6665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.479005 6511 server_base.cc:1034] running on GCE node
W20260808 02:17:56.479081 6668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.481002 6666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.481271 6511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:56.481434 6511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:56.482576 6511 hybrid_clock.cc:648] HybridClock initialized: now 1786155476482568 us; error 31 us; skew 500 ppm
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:56.484320 6511 init.cc:377] Logged in from keytab as oryx/127.28.177.188@KRBTEST.COM (short username oryx)
I20260808 02:17:56.484622 6511 webserver.cc:460] Webserver started at http://127.28.177.188:44717/ using document root <none> and password file <none>
I20260808 02:17:56.484745 6511 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.484786 6511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.485685 6511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:17:56.486356 6675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.486516 6511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:17:56.486573 6511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
uuid: "b11f30b06888407d9ef5377a1b6cd8e0"
format_stamp: "Formatted at 2026-08-08 02:17:56 on dist-test-slave-3h5h"
server_key: "fe1c7d5b1955655488b9424acddf3d68"
server_key_iv: "5191c6734a4ac6a83c437db24790e024"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.486650 6511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.513770 6511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:56.515892 6511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.519076 6511 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.188:33859
I20260808 02:17:56.519208 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.188:33859 every 8 connection(s)
I20260808 02:17:56.519448 6511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
I20260808 02:17:56.520448 6754 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:17:56.521020 6754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Bootstrap starting.
I20260808 02:17:56.524566 6754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:17:56.524818 6754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Bootstrap complete.
I20260808 02:17:56.525082 6754 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } }
I20260808 02:17:56.525197 6754 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b11f30b06888407d9ef5377a1b6cd8e0, State: Initialized, Role: LEARNER
I20260808 02:17:56.525240 6754 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } }
I20260808 02:17:56.525447 6754 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:17:56.527275 6773 catalog_manager.cc:1259] Loaded cluster ID: 5acafbffa24548d7bde6b3b6009abd63
I20260808 02:17:56.527410 6773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: loading cluster ID for follower catalog manager: success
I20260808 02:17:56.527611 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.528342 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:17:56.529707 6773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: acquiring CA information for follower catalog manager: success
I20260808 02:17:56.530059 6773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:17:56.547646 6648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:56.547868 6648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:56.547894 6648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:56.549257 6648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:56.549304 6648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:56.549335 6648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:17:56.551039 6648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:42297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=33529
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:56.551316 6648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:56.551608 6648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:56.558089 6780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.558205 6783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.558291 6648 server_base.cc:1034] running on GCE node
W20260808 02:17:56.558458 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.558671 6648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:56.558986 6648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:56.560177 6648 hybrid_clock.cc:648] HybridClock initialized: now 1786155476560149 us; error 45 us; skew 500 ppm
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:56.563203 6648 init.cc:377] Logged in from keytab as oryx/127.28.177.129@KRBTEST.COM (short username oryx)
I20260808 02:17:56.563719 6648 webserver.cc:460] Webserver started at http://127.28.177.129:33529/ using document root <none> and password file <none>
I20260808 02:17:56.563971 6648 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.564050 6648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.565569 6648 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:17:56.566407 6801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.566632 6648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:56.566766 6648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "63d8dcb828ef45e193672f5503fbf91e"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "aaabe2feca8c6436ce5dcf3b7c34d22c"
server_key_iv: "da647cb096a5eb4c18895003c5bf2e11"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.567122 6648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.584968 6648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:56.588099 6648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:56.588260 6648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.588500 6648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:56.588815 6648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:56.588945 6648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.588989 6648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:56.589012 6648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.596614 6648 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:42297
I20260808 02:17:56.596712 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:42297 every 8 connection(s)
I20260808 02:17:56.597160 6648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
I20260808 02:17:56.606060 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 6648
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:17:56.606254 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 5860
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.188@KRBTEST.COM
I20260808 02:17:56.612017 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.598476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:34137 (local address 127.28.177.190:42235)
0808 02:17:56.598591 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.598594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.598604 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:17:56.599186 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.599187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.599203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.599237 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.600364 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.601026 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.602329 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.602635 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.602676 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.607579 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.607581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.607581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.607591 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.609870 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.610508 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.610509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.610510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.610597 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.611078 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.611080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.611080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.611297 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.611362 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.611788 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.611898 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:17:56.612365 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.598290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:46867 (local address 127.28.177.189:38019)
0808 02:17:56.598353 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.598355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.598711 (+ 356us) server_negotiation.cc:407] Connection header received
0808 02:17:56.599176 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.599178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.599194 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.599288 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.600938 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.601749 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.602715 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.602948 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.603008 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.606080 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.606082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.606083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.606093 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.610571 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.611037 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.611038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.611039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.611121 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.611370 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.611371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.611371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.611533 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.611629 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.612222 (+ 593us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.612299 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 02:17:56.612617 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.598091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:34137)
0808 02:17:56.598382 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.598438 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.599042 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.599373 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.599377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.599922 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.600235 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.600241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.601156 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.601157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.602224 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.602226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.602773 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.602773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.602885 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.603611 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.603623 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.607453 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.610018 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.610030 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.610034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.610397 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.610709 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.610710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.610711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.610965 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.611416 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.611424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.611615 (+ 191us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.611735 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.612353 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":222,"thread_start_us":94,"threads_started":1}
I20260808 02:17:56.612637 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.598091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.28.177.129:46867)
0808 02:17:56.598620 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.598634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.599050 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.599739 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.599747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.600077 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.600234 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.600239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.601856 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.601857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.602601 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.602602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.603031 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.603032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.603100 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.603890 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.603896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.605935 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.610687 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.610688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.610688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.610916 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.611203 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.611204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.611205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.611287 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.611618 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.611620 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.611671 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.611784 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.611979 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":7,"thread_start_us":122,"threads_started":1}
I20260808 02:17:56.613245 6966 heartbeater.cc:347] Connected to a master server at 127.28.177.189:38019
I20260808 02:17:56.613358 6966 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.613566 6966 heartbeater.cc:511] Master 127.28.177.189:38019 requested a full tablet report, sending...
I20260808 02:17:56.614292 6424 ts_manager.cc:194] Registered new tserver with Master: 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129:42297)
I20260808 02:17:56.614390 6965 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42235
I20260808 02:17:56.614437 6965 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.614516 6965 heartbeater.cc:511] Master 127.28.177.190:42235 requested a full tablet report, sending...
I20260808 02:17:56.614893 5596 ts_manager.cc:194] Re-registered known tserver with Master: 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129:42297)
I20260808 02:17:56.615406 5596 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:34137
I20260808 02:17:56.619478 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:44521
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:42235
--webserver_port=43713
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:17:56.620618 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.598675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:46825 (local address 127.28.177.188:33859)
0808 02:17:56.598906 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.598908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.598917 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:56.599214 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.599215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.599229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.599443 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.600350 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.601124 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.602349 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.602602 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.602699 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.617333 (+ 14634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.617335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.617336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.617346 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.619316 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.619809 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.619810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.619810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.619872 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.620106 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.620107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.620107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.620224 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.620278 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.620472 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.620549 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
I20260808 02:17:56.620814 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.598453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.28.177.129:46825)
0808 02:17:56.598784 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.598796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.599054 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.599392 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.599392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.599922 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.600236 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.600241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.601205 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.601206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.602228 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.602231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.603068 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.603069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.603171 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.604966 (+ 1795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.604972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.617219 (+ 12247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.619410 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.619411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.619412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.619709 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.619945 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.619946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.619946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.620036 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.620294 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.620295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.620350 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.620445 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.620624 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":34,"thread_start_us":73,"threads_started":1}
I20260808 02:17:56.621340 6964 heartbeater.cc:347] Connected to a master server at 127.28.177.188:33859
I20260808 02:17:56.621415 6964 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.621526 6964 heartbeater.cc:511] Master 127.28.177.188:33859 requested a full tablet report, sending...
I20260808 02:17:56.621992 6702 ts_manager.cc:194] Registered new tserver with Master: 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129:42297)
W20260808 02:17:56.707777 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:56.707999 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:56.708024 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:56.709169 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:56.709208 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:56.709233 6973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:17:56.710659 6973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:44521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=43713
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:56.710848 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:56.711042 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:56.716645 6981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.716686 6980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.716615 6983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.716964 6973 server_base.cc:1034] running on GCE node
I20260808 02:17:56.717258 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:56.717480 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:56.718681 6973 hybrid_clock.cc:648] HybridClock initialized: now 1786155476718666 us; error 48 us; skew 500 ppm
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:56.721642 6973 init.cc:377] Logged in from keytab as oryx/127.28.177.130@KRBTEST.COM (short username oryx)
I20260808 02:17:56.722209 6973 webserver.cc:460] Webserver started at http://127.28.177.130:43713/ using document root <none> and password file <none>
I20260808 02:17:56.722451 6973 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.722543 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.724074 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:56.724735 6993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.724980 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:56.725164 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "de1cb9d1f041494090fb3653a8bdb275"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "fcce6a72b71cf84aa0b863e90cab9620"
server_key_iv: "7c3e839d5d73fd540a3f30baa0569e73"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.725602 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.744284 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:56.746888 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:56.747083 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.747363 6973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:56.747717 6973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:56.747774 6973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.747817 6973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:56.747845 6973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.754060 6973 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:44521
I20260808 02:17:56.754104 7152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:44521 every 8 connection(s)
I20260808 02:17:56.754539 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:17:56.756480 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 6973
I20260808 02:17:56.756654 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 6046
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:17:56.765255 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.756073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37319 (local address 127.28.177.189:38019)
0808 02:17:56.756142 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.756144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.756160 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:17:56.756450 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.756452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.756466 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.756501 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.757435 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.758099 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.759290 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.759497 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.759534 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.762157 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.762159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.762159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.762167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.763800 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.764262 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.764264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.764264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.764318 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.764583 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.764583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.764584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.764704 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.764742 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.765095 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.765172 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:17:56.765756 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.755882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.28.177.130:37319)
0808 02:17:56.756043 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.756054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.756320 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.756709 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.756713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.757077 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.757335 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.757340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.758220 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.758220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.759202 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.759204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.759610 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.759611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.759723 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.760244 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.760253 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.762055 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.763885 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.763895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.763897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.764177 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.764378 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.764378 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.764378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.764518 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.764778 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.764783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.764956 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.765382 (+ 426us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.765552 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260808 02:17:56.767082 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.755761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:57919 (local address 127.28.177.190:42235)
0808 02:17:56.755810 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.755812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.755860 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:17:56.756469 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.756471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.756487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.756521 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.757559 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.758606 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.760012 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.760417 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.760468 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.763188 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.763190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.763190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.763201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.765392 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.765935 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.765936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.765937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.766012 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.766283 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.766284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.766284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.766469 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.766527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.766816 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.766926 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:17:56.767524 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.755665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.130:57919)
0808 02:17:56.755800 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.755810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.756335 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.756710 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.756714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.757353 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.757436 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.757438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.758737 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.758738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.759889 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.759893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.760559 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.760560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.760642 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.761081 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.761087 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.763053 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.765511 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.765512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.765513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.765828 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.766106 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.766107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.766108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.766183 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.766589 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.766590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.766659 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.767197 (+ 538us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.767347 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260808 02:17:56.768095 7155 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42235
I20260808 02:17:56.768227 7155 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.768503 7155 heartbeater.cc:511] Master 127.28.177.190:42235 requested a full tablet report, sending...
I20260808 02:17:56.777506 5596 ts_manager.cc:194] Re-registered known tserver with Master: de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521)
I20260808 02:17:56.778059 5596 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:57919
I20260808 02:17:56.778594 7157 heartbeater.cc:347] Connected to a master server at 127.28.177.189:38019
I20260808 02:17:56.778690 7157 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.778798 7157 heartbeater.cc:511] Master 127.28.177.189:38019 requested a full tablet report, sending...
I20260808 02:17:56.779151 6424 ts_manager.cc:194] Registered new tserver with Master: de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521)
I20260808 02:17:56.782742 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:43219
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:42235
--webserver_port=43411
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.188@KRBTEST.COM
I20260808 02:17:56.787729 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.755380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37583 (local address 127.28.177.188:33859)
0808 02:17:56.755450 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.755452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.755720 (+ 268us) server_negotiation.cc:407] Connection header received
0808 02:17:56.756450 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.756452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.756467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.756510 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.779654 (+ 23144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.780492 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.781472 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.781847 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.781938 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.784498 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.784500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.784501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.784511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.786267 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.786762 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.786763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.786763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.786822 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.787100 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.787101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.787101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.787225 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.787283 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.787500 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.787608 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:17:56.787847 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.755392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.28.177.130:37583)
0808 02:17:56.755556 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.755593 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.756320 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.779243 (+ 22923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.779244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.779474 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.779532 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.779534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.780610 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.780611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.781363 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.781366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.781941 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.781942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.781999 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.782625 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.782631 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.784355 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.786368 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.786370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.786371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.786676 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.786906 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.786907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.786908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.787017 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.787306 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.787308 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.787371 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.787480 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.787632 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260808 02:17:56.788323 7153 heartbeater.cc:347] Connected to a master server at 127.28.177.188:33859
I20260808 02:17:56.788408 7153 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.788532 7153 heartbeater.cc:511] Master 127.28.177.188:33859 requested a full tablet report, sending...
I20260808 02:17:56.789002 6702 ts_manager.cc:194] Registered new tserver with Master: de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521)
I20260808 02:17:56.843106 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.839459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.28.177.188:33859)
0808 02:17:56.839510 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.839515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.839530 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:17:56.839646 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.839647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.839674 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.839761 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:56.839993 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.840610 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.842235 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.842664 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.842739 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.842800 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.842948 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.842979 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:17:56.843098 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.839374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55616)
0808 02:17:56.839449 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.839464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.839521 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.839798 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.839800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.839802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:56.839867 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.839870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.840715 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.840716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.842060 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.842064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.842767 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.842769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.842847 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.842852 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.842945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.842998 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 02:17:56.843680 6719 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 02:17:56.843915 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50fe97b61719474d80bf4b8325421662. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.844048 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: This master's current role is: LEARNER
W20260808 02:17:56.882848 7163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:17:56.883173 7163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:17:56.883224 7163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:17:56.884565 7163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:17:56.884649 7163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:17:56.884703 7163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:17:56.886601 7163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:43219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=43411
--tserver_master_addrs=127.28.177.190:42235,127.28.177.189:38019,127.28.177.188:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:17:56.886932 7163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:17:56.887240 7163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:17:56.892915 7169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.892921 7170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:17:56.892983 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:17:56.893049 7163 server_base.cc:1034] running on GCE node
I20260808 02:17:56.893366 7163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:17:56.893627 7163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:17:56.894814 7163 hybrid_clock.cc:648] HybridClock initialized: now 1786155476894812 us; error 40 us; skew 500 ppm
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:17:56.897478 7163 init.cc:377] Logged in from keytab as oryx/127.28.177.131@KRBTEST.COM (short username oryx)
I20260808 02:17:56.897945 7163 webserver.cc:460] Webserver started at http://127.28.177.131:43411/ using document root <none> and password file <none>
I20260808 02:17:56.898257 7163 fs_manager.cc:359] Metadata directory not provided
I20260808 02:17:56.898362 7163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:17:56.899864 7163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 02:17:56.900588 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.900792 7163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:17:56.900918 7163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "8fe525ab07cd4df1a2738e4f62f37db4"
format_stamp: "Formatted at 2026-08-08 02:17:55 on dist-test-slave-3h5h"
server_key: "717e5229dc4561282df022b136e5dceb"
server_key_iv: "61d60db596db681280c636817f50d9e3"
server_key_version: "encryptionkey@0"
I20260808 02:17:56.901347 7163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:17:56.920468 7163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:17:56.922605 7163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:17:56.922780 7163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:17:56.923043 7163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:17:56.923394 7163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:17:56.923451 7163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.923492 7163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:17:56.923523 7163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:17:56.930719 7163 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:43219
I20260808 02:17:56.930766 7361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:43219 every 8 connection(s)
I20260808 02:17:56.931164 7163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.188@KRBTEST.COM
I20260808 02:17:56.939756 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 7163
Aug 08 02:17:56 dist-test-slave-3h5h krb5kdc[5540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155476, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.189@KRBTEST.COM
I20260808 02:17:56.943763 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.932052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:41803 (local address 127.28.177.189:38019)
0808 02:17:56.932201 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.932204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.932399 (+ 195us) server_negotiation.cc:407] Connection header received
0808 02:17:56.933418 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.933419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.933433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.933469 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.934492 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.935327 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936624 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.936954 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.937038 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.940442 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.940443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.940443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.940452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.942009 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942553 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.942555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.942555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.942619 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942933 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.942934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.942934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.943100 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.943151 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.943564 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.943660 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:17:56.944024 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.932624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.131:56925)
0808 02:17:56.932757 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.932768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.933294 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.933925 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.933926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.934139 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.934367 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.934371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.935100 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.935101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.935958 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.935961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.936364 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.936495 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.937252 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.937261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.940177 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.942298 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.942300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.942545 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.942760 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.942761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.942856 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.943223 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.943224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.943404 (+ 180us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.943516 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.943728 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260808 02:17:56.944320 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.932508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:36645 (local address 127.28.177.188:33859)
0808 02:17:56.932559 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.932562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.932697 (+ 135us) server_negotiation.cc:407] Connection header received
0808 02:17:56.933555 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.933557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.933573 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.933637 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.934500 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.935339 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936598 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.936916 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936995 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.940014 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.940016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.940016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.940024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.942818 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.943250 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.943251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.943252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.943312 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.943509 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.943510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.943510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.943638 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.943700 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.944111 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.944221 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:17:56.944885 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.932726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:56925 (local address 127.28.177.190:42235)
0808 02:17:56.932790 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.932793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.932804 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:56.933412 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.933414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.933428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.933494 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.934473 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.934985 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936055 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.936262 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936352 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.940286 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.940287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.940287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.940296 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.942189 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942610 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.942611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.942612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.942683 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942935 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.942936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.942937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.943113 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.943179 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.944659 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.944745 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:17:56.944957 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.932426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.28.177.131:36645)
0808 02:17:56.932599 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.932611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.933287 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.933665 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.933666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.934149 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.934368 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.934374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.935440 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.935441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.936501 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.936504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.936994 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.936995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.937059 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.937503 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.937508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.939899 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.942933 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.942934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.943158 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.943380 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.943381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.943381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.943436 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.943727 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.943728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.943765 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.943830 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.943967 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":79,"thread_start_us":83,"threads_started":1}
I20260808 02:17:56.945255 7364 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42235
I20260808 02:17:56.945334 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.932036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.28.177.131:41803)
0808 02:17:56.932239 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.932297 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.933279 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.933570 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.933574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.934142 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:17:56.934368 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.934374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.935440 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.935441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.936501 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.936504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.937105 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.937106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.937216 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.937712 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:17:56.937724 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:17:56.940341 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:17:56.942108 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.942118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.942455 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.942711 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:17:56.942712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:17:56.942713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:17:56.942853 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:17:56.943215 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:17:56.943222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:17:56.943403 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.943884 (+ 481us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.944042 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":151,"thread_start_us":79,"threads_started":1}
I20260808 02:17:56.945382 7364 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.945549 7363 heartbeater.cc:347] Connected to a master server at 127.28.177.188:33859
I20260808 02:17:56.945597 7364 heartbeater.cc:511] Master 127.28.177.190:42235 requested a full tablet report, sending...
I20260808 02:17:56.945598 7363 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.945685 7363 heartbeater.cc:511] Master 127.28.177.188:33859 requested a full tablet report, sending...
I20260808 02:17:56.945997 7365 heartbeater.cc:347] Connected to a master server at 127.28.177.189:38019
I20260808 02:17:56.946050 7365 heartbeater.cc:464] Registering TS with master...
I20260808 02:17:56.946120 7365 heartbeater.cc:511] Master 127.28.177.189:38019 requested a full tablet report, sending...
I20260808 02:17:56.946112 6702 ts_manager.cc:194] Registered new tserver with Master: 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:56.946364 5596 ts_manager.cc:194] Re-registered known tserver with Master: 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:56.946410 6424 ts_manager.cc:194] Registered new tserver with Master: 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:56.946976 5596 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:56925
I20260808 02:17:56.953560 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.942786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.28.177.190:42235)
0808 02:17:56.944962 (+ 2176us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.944964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.946264 (+ 1300us) server_negotiation.cc:407] Connection header received
0808 02:17:56.946366 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.946368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.946379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.946408 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.947043 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.947646 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.949013 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.949313 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.949381 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.950758 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.950759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.950760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.950768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.952275 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.952696 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.952697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.952697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.952763 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.953025 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.953026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.953027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.953142 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.953252 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.953365 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.953444 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2161,"thread_start_us":82,"threads_started":1}
I20260808 02:17:56.954156 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.942107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.28.177.189:38019)
0808 02:17:56.942410 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.942412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.943068 (+ 656us) server_negotiation.cc:407] Connection header received
0808 02:17:56.943107 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.943107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.943123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.943180 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.944438 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.946194 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.947784 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.948094 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.948157 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.949628 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.949629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.949630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.949637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.951673 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.952112 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.952114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.952115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.952178 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.952463 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.952464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.952464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.952952 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.953071 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.953953 (+ 882us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.954057 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":66,"threads_started":1}
I20260808 02:17:56.955935 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.943508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.28.177.188:33859)
0808 02:17:56.944387 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.944389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.947984 (+ 3595us) server_negotiation.cc:407] Connection header received
0808 02:17:56.948052 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.948053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.948064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.948113 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:56.948666 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.949405 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.950431 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.950767 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.950833 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.952396 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:56.952397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:56.952398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:56.952406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:56.954627 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.955077 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.955078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.955078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.955145 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:56.955393 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:56.955394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:56.955394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:56.955537 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:56.955645 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.955753 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.955830 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":60,"threads_started":1}
I20260808 02:17:56.955755 5596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:56.957779 5596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:56.958644 6651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: NON_VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: true } } }
I20260808 02:17:56.958765 6651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.959957 7391 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: attempting to promote NON_VOTER b11f30b06888407d9ef5377a1b6cd8e0 to VOTER
I20260808 02:17:56.960129 7391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } }
I20260808 02:17:56.960697 6437 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Refusing update from remote peer 50fe97b61719474d80bf4b8325421662: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 02:17:56.960980 7391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 02:17:56.961079 6719 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 LEARNER]: Refusing update from remote peer 50fe97b61719474d80bf4b8325421662: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 02:17:56.961383 7391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 02:17:56.961829 6651 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.961956 6437 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.962452 6719 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.962900 6651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.962968 6651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.963016 6651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.963055 6651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.963047 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.963177 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:56.963479 7402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.963572 7402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:17:56.963554 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50fe97b61719474d80bf4b8325421662" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:56.963644 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:17:56.967795 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.963783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44521 (local address 127.0.0.1:35268)
0808 02:17:56.963820 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.963831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.963871 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.964752 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.964753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.964755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:56.964800 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.964801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.965937 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.965938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.966847 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.966849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.967563 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.967565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.967623 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.967627 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.967673 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.967715 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":3}
I20260808 02:17:56.967806 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.964337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.28.177.130:44521)
0808 02:17:56.964581 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.964584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.964593 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:56.964624 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.964625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.964638 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.964739 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:56.964904 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.965838 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.966945 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.967472 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.967585 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.967661 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.967690 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.967728 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":140,"threads_started":1}
I20260808 02:17:56.968288 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.963876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42297 (local address 127.0.0.1:57360)
0808 02:17:56.964001 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.964010 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.964050 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.964431 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.964432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.964435 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:56.964494 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.964496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.965978 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.965979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.967124 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.967127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.968058 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.968059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.968123 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.968128 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.968160 (+ 32us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.968202 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":87,"threads_started":1}
I20260808 02:17:56.968780 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.964073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:43219 (local address 127.0.0.1:41988)
0808 02:17:56.964730 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:56.964741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:56.964797 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:56.965096 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:56.965097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:56.965099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:56.965159 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.965160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.966390 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.966391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.967457 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:56.967459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.968507 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.968508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:56.968574 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.968579 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:17:56.968636 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:17:56.968688 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":89,"threads_started":1}
I20260808 02:17:56.968763 7056 tablet_service.cc:1467] Processing CreateTablet for tablet b6eedf4888fc4336bf09a68ee3ec3dab (DEFAULT_TABLE table=test-table [id=e6410d0faa40410fb2a44847f8b811cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:56.969017 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.964203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.28.177.131:43219)
0808 02:17:56.964467 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.964472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.964887 (+ 415us) server_negotiation.cc:407] Connection header received
0808 02:17:56.964936 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.964939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.964957 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.965027 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:56.965234 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.966296 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.967546 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.968310 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.968403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.968464 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.968892 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.968938 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
I20260808 02:17:56.969031 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:56.963999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.28.177.129:42297)
0808 02:17:56.964233 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:56.964239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:56.964250 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:17:56.964296 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:56.964298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:56.964315 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:56.964578 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:56.964667 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.965829 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.967230 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:56.967956 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:56.968754 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:56.968822 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:56.968912 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 02:17:56.968957 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
I20260808 02:17:56.969275 7056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6eedf4888fc4336bf09a68ee3ec3dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:56.972595 7409 tablet_bootstrap.cc:492] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Bootstrap starting.
I20260808 02:17:56.973378 7409 tablet_bootstrap.cc:654] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:56.973443 6871 tablet_service.cc:1467] Processing CreateTablet for tablet b6eedf4888fc4336bf09a68ee3ec3dab (DEFAULT_TABLE table=test-table [id=e6410d0faa40410fb2a44847f8b811cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:56.973438 7261 tablet_service.cc:1467] Processing CreateTablet for tablet b6eedf4888fc4336bf09a68ee3ec3dab (DEFAULT_TABLE table=test-table [id=e6410d0faa40410fb2a44847f8b811cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:56.973770 7409 log.cc:824] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:56.973812 6871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6eedf4888fc4336bf09a68ee3ec3dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:56.973814 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6eedf4888fc4336bf09a68ee3ec3dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:56.974586 7409 tablet_bootstrap.cc:492] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: No bootstrap required, opened a new log
I20260808 02:17:56.974678 7409 ts_tablet_manager.cc:1397] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:56.976925 7409 raft_consensus.cc:348] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:56.977089 7409 raft_consensus.cc:374] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:56.977140 7409 raft_consensus.cc:729] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1cb9d1f041494090fb3653a8bdb275, State: Initialized, Role: FOLLOWER
I20260808 02:17:56.977279 7409 consensus_queue.cc:260] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:56.977576 7409 ts_tablet_manager.cc:1428] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:17:56.977726 7155 heartbeater.cc:503] Master 127.28.177.190:42235 was elected leader, sending a full tablet report...
I20260808 02:17:56.977763 7414 tablet_bootstrap.cc:492] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Bootstrap starting.
I20260808 02:17:56.977594 7416 tablet_bootstrap.cc:492] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Bootstrap starting.
I20260808 02:17:56.978682 7414 tablet_bootstrap.cc:654] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:56.979385 7416 tablet_bootstrap.cc:654] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:56.979740 7416 log.cc:824] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:56.980221 7414 log.cc:824] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:17:56.980515 7416 tablet_bootstrap.cc:492] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: No bootstrap required, opened a new log
I20260808 02:17:56.980598 7416 ts_tablet_manager.cc:1397] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:17:56.981038 7414 tablet_bootstrap.cc:492] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: No bootstrap required, opened a new log
I20260808 02:17:56.981096 7414 ts_tablet_manager.cc:1397] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:17:56.982494 7414 raft_consensus.cc:348] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:56.982632 7414 raft_consensus.cc:374] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:56.982656 7414 raft_consensus.cc:729] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d8dcb828ef45e193672f5503fbf91e, State: Initialized, Role: FOLLOWER
I20260808 02:17:56.982575 7416 raft_consensus.cc:348] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:56.982745 7416 raft_consensus.cc:374] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:56.982790 7416 raft_consensus.cc:729] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe525ab07cd4df1a2738e4f62f37db4, State: Initialized, Role: FOLLOWER
I20260808 02:17:56.982806 7414 consensus_queue.cc:260] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:56.982908 7416 consensus_queue.cc:260] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:56.983108 7414 ts_tablet_manager.cc:1428] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:17:56.983208 7416 ts_tablet_manager.cc:1428] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:17:56.983367 7364 heartbeater.cc:503] Master 127.28.177.190:42235 was elected leader, sending a full tablet report...
I20260808 02:17:56.983371 6965 heartbeater.cc:503] Master 127.28.177.190:42235 was elected leader, sending a full tablet report...
W20260808 02:17:57.005326 7159 tablet.cc:2369] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:17:57.098376 6967 tablet.cc:2369] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:17:57.127383 7420 raft_consensus.cc:482] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:57.127523 7420 raft_consensus.cc:504] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:57.127889 7420 leader_election.cc:288] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521), 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:57.131392 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.128225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:57999 (local address 127.28.177.130:44521)
0808 02:17:57.128306 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:57.128308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:57.128469 (+ 161us) server_negotiation.cc:407] Connection header received
0808 02:17:57.128625 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:57.128626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:57.128636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:57.128691 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:57.128961 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.129437 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.130580 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.130991 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.131087 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.131098 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:57.131262 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 02:17:57.131305 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:17:57.131408 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.128474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:57875 (local address 127.28.177.131:43219)
0808 02:17:57.128529 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:57.128531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:57.128540 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:57.128618 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:57.128619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:57.128629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:57.128706 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:57.128942 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.129541 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.130580 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.130975 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.131103 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.131113 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:57.131295 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:17:57.131336 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:17:57.131455 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.128368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:43219 (local address 127.28.177.129:57875)
0808 02:17:57.128463 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:57.128476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:57.128535 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:57.128713 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:57.128714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:57.128716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:57.128782 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:57.128789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.129637 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.129638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:57.130471 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:57.130473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.131116 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.131117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:57.131192 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.131195 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:57.131284 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:17:57.131330 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":39,"threads_started":1}
I20260808 02:17:57.131470 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.128166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44521 (local address 127.28.177.129:57999)
0808 02:17:57.128340 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:57.128367 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:57.128478 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:57.128748 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:57.128749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:57.128750 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:57.128814 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:57.128815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.129529 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.129530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:57.130470 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:57.130473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.131087 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.131088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:57.131163 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.131167 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:57.131253 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:17:57.131305 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":55,"thread_start_us":76,"threads_started":1}
I20260808 02:17:57.131992 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eedf4888fc4336bf09a68ee3ec3dab" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" is_pre_election: true
I20260808 02:17:57.132059 7079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eedf4888fc4336bf09a68ee3ec3dab" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1cb9d1f041494090fb3653a8bdb275" is_pre_election: true
I20260808 02:17:57.132177 7079 raft_consensus.cc:2454] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 0.
I20260808 02:17:57.132176 7290 raft_consensus.cc:2454] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 0.
I20260808 02:17:57.132436 6802 leader_election.cc:302] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d8dcb828ef45e193672f5503fbf91e, de1cb9d1f041494090fb3653a8bdb275; no voters:
I20260808 02:17:57.132613 7420 raft_consensus.cc:2781] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:57.132730 7420 raft_consensus.cc:482] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:57.132772 7420 raft_consensus.cc:3037] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:57.133634 7420 raft_consensus.cc:504] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:57.133822 7420 leader_election.cc:288] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 election: Requested vote from peers de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521), 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:57.134044 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eedf4888fc4336bf09a68ee3ec3dab" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe525ab07cd4df1a2738e4f62f37db4"
I20260808 02:17:57.134138 7290 raft_consensus.cc:3037] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:57.134099 7079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6eedf4888fc4336bf09a68ee3ec3dab" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1cb9d1f041494090fb3653a8bdb275"
I20260808 02:17:57.134230 7079 raft_consensus.cc:3037] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:57.134917 7290 raft_consensus.cc:2454] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 1.
I20260808 02:17:57.135097 7079 raft_consensus.cc:2454] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 1.
I20260808 02:17:57.135115 6802 leader_election.cc:302] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d8dcb828ef45e193672f5503fbf91e, 8fe525ab07cd4df1a2738e4f62f37db4; no voters:
I20260808 02:17:57.135260 7420 raft_consensus.cc:2781] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:57.135423 7420 raft_consensus.cc:686] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 1 LEADER]: Becoming Leader. State: Replica: 63d8dcb828ef45e193672f5503fbf91e, State: Running, Role: LEADER
I20260808 02:17:57.135596 7420 consensus_queue.cc:237] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } }
I20260808 02:17:57.136669 5596 catalog_manager.cc:5495] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129). New cstate: current_term: 1 leader_uuid: "63d8dcb828ef45e193672f5503fbf91e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } health_report { overall_health: UNKNOWN } } }
I20260808 02:17:57.149856 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.147304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.28.177.129:42297)
0808 02:17:57.147381 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:57.147383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:57.147391 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:57.147431 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:57.147432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:57.147449 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:57.147526 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:57.147703 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.148220 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.149061 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.149262 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.149339 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.149515 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:57.149627 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:57.149729 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:57.149747 (+ 18us) server_negotiation.cc:299] Negotiation successful
0808 02:17:57.149769 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:17:57.151985 7079 raft_consensus.cc:1261] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Refusing update from remote peer 63d8dcb828ef45e193672f5503fbf91e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:57.151985 7290 raft_consensus.cc:1261] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Refusing update from remote peer 63d8dcb828ef45e193672f5503fbf91e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:57.152407 7420 consensus_queue.cc:1035] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:57.152544 7426 consensus_queue.cc:1035] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:57.155694 7431 mvcc.cc:205] Tried to move back new op lower bound from 7316092834411241472 to 7316092834348085248. Current Snapshot: MvccSnapshot[applied={T|T < 7316092834411241472}]
I20260808 02:17:57.156639 7434 mvcc.cc:205] Tried to move back new op lower bound from 7316092834411241472 to 7316092834348085248. Current Snapshot: MvccSnapshot[applied={T|T < 7316092834411241472 or (T in {7316092834411241472})}]
I20260808 02:17:57.158008 7432 mvcc.cc:205] Tried to move back new op lower bound from 7316092834411241472 to 7316092834348085248. Current Snapshot: MvccSnapshot[applied={T|T < 7316092834411241472 or (T in {7316092834411241472})}]
I20260808 02:17:57.158412 5595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:57.158522 5595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:57.160001 5595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50fe97b61719474d80bf4b8325421662: Sending DeleteTablet for 3 replicas of tablet b6eedf4888fc4336bf09a68ee3ec3dab
I20260808 02:17:57.160758 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet b6eedf4888fc4336bf09a68ee3ec3dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:57 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:41988
I20260808 02:17:57.160773 7056 tablet_service.cc:1514] Processing DeleteTablet for tablet b6eedf4888fc4336bf09a68ee3ec3dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:57 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:35268
I20260808 02:17:57.161120 6871 tablet_service.cc:1514] Processing DeleteTablet for tablet b6eedf4888fc4336bf09a68ee3ec3dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:57 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:57360
I20260808 02:17:57.161559 7447 tablet_replica.cc:330] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: stopping tablet replica
I20260808 02:17:57.161706 7447 raft_consensus.cc:2229] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:57.161713 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 5560
I20260808 02:17:57.167968 7447 raft_consensus.cc:2258] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:57.168967 7447 ts_tablet_manager.cc:1905] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:57.171468 7447 ts_tablet_manager.cc:1918] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:57.171559 7447 log.cc:1196] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/b6eedf4888fc4336bf09a68ee3ec3dab
I20260808 02:17:57.171872 7447 ts_tablet_manager.cc:1939] T b6eedf4888fc4336bf09a68ee3ec3dab P de1cb9d1f041494090fb3653a8bdb275: Deleting consensus metadata
W20260808 02:17:57.176123 6998 connection.cc:535] server connection from 127.0.0.1:35268 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:17:57.176688 7445 tablet_replica.cc:330] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: stopping tablet replica
I20260808 02:17:57.176777 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.160943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.28.177.188:33859)
0808 02:17:57.160988 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:57.160991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:57.168340 (+ 7349us) server_negotiation.cc:407] Connection header received
0808 02:17:57.168431 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:57.168432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:57.168446 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:57.168518 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:57.169449 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.170288 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.171427 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.171721 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.171767 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.173563 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:57.173564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:57.173565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:57.173575 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:57.175433 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:57.175863 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:57.175864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:57.175865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:57.175927 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:57.176129 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:57.176130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:57.176130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:57.176280 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:57.176369 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:57.176591 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:17:57.176679 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:17:57.176828 7445 raft_consensus.cc:2229] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:57.177004 7445 raft_consensus.cc:2258] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:57.177219 7450 tablet_replica.cc:330] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: stopping tablet replica
I20260808 02:17:57.177332 7450 raft_consensus.cc:2229] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:17:57.177440 7450 raft_consensus.cc:2258] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:57.178678 7445 ts_tablet_manager.cc:1905] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:57.180706 7445 ts_tablet_manager.cc:1918] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:57.180784 7445 log.cc:1196] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/b6eedf4888fc4336bf09a68ee3ec3dab
I20260808 02:17:57.181092 7445 ts_tablet_manager.cc:1939] T b6eedf4888fc4336bf09a68ee3ec3dab P 8fe525ab07cd4df1a2738e4f62f37db4: Deleting consensus metadata
W20260808 02:17:57.181459 7188 connection.cc:441] server connection from 127.0.0.1:41988 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41988 (request call id 1) could send its response
I20260808 02:17:57.186702 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.178359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.28.177.188:33859)
0808 02:17:57.178399 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:57.178401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:57.178661 (+ 260us) server_negotiation.cc:407] Connection header received
0808 02:17:57.178758 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:57.178759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:57.178769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:57.178819 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:57.179838 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.180640 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.181737 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.182079 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.182121 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.183591 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:57.183593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:57.183593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:57.183600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:57.185321 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:57.185759 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:57.185760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:57.185761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:57.185821 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:57.186068 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:57.186069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:57.186070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:57.186209 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:57.186268 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:57.186490 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:17:57.186602 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:17:57.186878 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:57.178236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.28.177.189:38019)
0808 02:17:57.178285 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:57.178288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:57.178489 (+ 201us) server_negotiation.cc:407] Connection header received
0808 02:17:57.178572 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:57.178573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:57.178591 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:57.178661 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:17:57.179322 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.180211 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.181630 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:57.181971 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:57.182045 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:57.183743 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:17:57.183744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:17:57.183744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:17:57.183753 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:17:57.185442 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:57.185860 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:57.185862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:57.185862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:57.185928 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:17:57.186134 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:17:57.186135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:17:57.186135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:17:57.186287 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:17:57.186384 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:57.186670 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 02:17:57.186777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:17:57.189548 7450 ts_tablet_manager.cc:1905] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:57.191548 7450 ts_tablet_manager.cc:1918] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:57.191648 7450 log.cc:1196] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/b6eedf4888fc4336bf09a68ee3ec3dab
I20260808 02:17:57.191972 7450 ts_tablet_manager.cc:1939] T b6eedf4888fc4336bf09a68ee3ec3dab P 63d8dcb828ef45e193672f5503fbf91e: Deleting consensus metadata
W20260808 02:17:57.192368 6810 connection.cc:441] server connection from 127.0.0.1:57360 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57360 (request call id 1) could send its response
I20260808 02:17:58.158126 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.157743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.130:35129)
0808 02:17:58.157974 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.158066 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":66,"threads_started":1}
W20260808 02:17:58.158358 7155 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:42235 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
I20260808 02:17:58.158521 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.157984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:57019)
0808 02:17:58.158137 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.158477 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260808 02:17:58.158550 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.157878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.131:46851)
0808 02:17:58.158438 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.158506 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":58,"threads_started":1}
W20260808 02:17:58.158742 7364 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:42235 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
W20260808 02:17:58.158743 6965 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:42235 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
I20260808 02:17:58.660137 7463 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50fe97b61719474d80bf4b8325421662)
I20260808 02:17:58.660287 7463 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } }
I20260808 02:17:58.660529 7463 leader_election.cc:288] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235), f6970c77282b4a55aaf31075a30ba57f (127.28.177.189:38019)
I20260808 02:17:58.661023 7468 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50fe97b61719474d80bf4b8325421662)
I20260808 02:17:58.661093 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.660751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35186)
0808 02:17:58.660905 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.660969 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":54,"threads_started":1}
I20260808 02:17:58.661162 7468 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } }
W20260808 02:17:58.661252 6678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
I20260808 02:17:58.661463 7468 leader_election.cc:288] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235), b11f30b06888407d9ef5377a1b6cd8e0 (127.28.177.188:33859)
I20260808 02:17:58.661731 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.661542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35192)
0808 02:17:58.661641 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.661691 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
I20260808 02:17:58.661721 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.661552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35194)
0808 02:17:58.661621 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.661686 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:17:58.661821 6678 leader_election.cc:334] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235): Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
W20260808 02:17:58.661875 6404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
I20260808 02:17:58.662384 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.662253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35198)
0808 02:17:58.662313 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.662357 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:17:58.662494 6404 leader_election.cc:334] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235): Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
I20260808 02:17:58.664556 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.660770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:38019 (local address 127.0.0.1:48638)
0808 02:17:58.661014 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.661027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:58.661238 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:58.661560 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:58.661562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:58.661565 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:58.661628 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.661631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.662778 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.662779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.663619 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.663622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.664268 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.664269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.664349 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.664353 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:58.664426 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:17:58.664473 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":45,"threads_started":1}
I20260808 02:17:58.664566 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.660901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.28.177.189:38019)
0808 02:17:58.661091 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:58.661094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:58.661106 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:17:58.661391 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:58.661392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:58.661407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:58.661487 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:58.661714 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.662701 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.663734 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.664140 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.664266 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.664277 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:58.664433 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:17:58.664504 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":49,"threads_started":1}
I20260808 02:17:58.664851 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6970c77282b4a55aaf31075a30ba57f" is_pre_election: true
I20260808 02:17:58.664963 6437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b11f30b06888407d9ef5377a1b6cd8e0 in term 1.
I20260808 02:17:58.665143 6676 leader_election.cc:302] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b11f30b06888407d9ef5377a1b6cd8e0, f6970c77282b4a55aaf31075a30ba57f; no voters: 50fe97b61719474d80bf4b8325421662
I20260808 02:17:58.665279 7463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 02:17:58.665339 7463 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 50fe97b61719474d80bf4b8325421662)
I20260808 02:17:58.665371 7463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:17:58.666150 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.661683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:33859 (local address 127.0.0.1:55650)
0808 02:17:58.661904 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.661914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:58.661966 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:58.662635 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:58.662637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:58.662640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:58.662693 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.662695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.664213 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.664214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.665283 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.665286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.665881 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.665883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.665947 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.665951 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:58.666036 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:17:58.666079 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":57,"threads_started":1}
I20260808 02:17:58.666152 7463 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } }
I20260808 02:17:58.666154 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.662088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.28.177.188:33859)
0808 02:17:58.662320 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:58.662321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:58.662328 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:17:58.662347 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:58.662348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:58.662355 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:58.662626 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:58.663709 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.664125 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.665399 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.665795 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.665885 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.665893 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:58.666045 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 02:17:58.666087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260808 02:17:58.666476 7463 leader_election.cc:288] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [CANDIDATE]: Term 2 election: Requested vote from peers 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235), f6970c77282b4a55aaf31075a30ba57f (127.28.177.189:38019)
I20260808 02:17:58.666445 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6970c77282b4a55aaf31075a30ba57f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" is_pre_election: true
I20260808 02:17:58.666535 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.666419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35214)
0808 02:17:58.666474 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.666508 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 02:17:58.666561 6719 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6970c77282b4a55aaf31075a30ba57f in current term 2: Already voted for candidate b11f30b06888407d9ef5377a1b6cd8e0 in this term.
I20260808 02:17:58.666615 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6970c77282b4a55aaf31075a30ba57f"
I20260808 02:17:58.666689 6437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:17:58.666736 6404 leader_election.cc:302] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6970c77282b4a55aaf31075a30ba57f; no voters: 50fe97b61719474d80bf4b8325421662, b11f30b06888407d9ef5377a1b6cd8e0
I20260808 02:17:58.667328 6437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b11f30b06888407d9ef5377a1b6cd8e0 in term 2.
I20260808 02:17:58.667531 7468 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:17:58.667526 6676 leader_election.cc:302] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b11f30b06888407d9ef5377a1b6cd8e0, f6970c77282b4a55aaf31075a30ba57f; no voters:
I20260808 02:17:58.667642 7463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:17:58.667764 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.667603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35216)
0808 02:17:58.667681 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.667740 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 02:17:58.667832 6678 leader_election.cc:334] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235): Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
I20260808 02:17:58.667876 7463 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [term 2 LEADER]: Becoming Leader. State: Replica: b11f30b06888407d9ef5377a1b6cd8e0, State: Running, Role: LEADER
I20260808 02:17:58.667991 7463 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } }
I20260808 02:17:58.668349 7484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11f30b06888407d9ef5377a1b6cd8e0. Latest consensus state: current_term: 2 leader_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:58.668450 7484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:58.668670 7487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:17:58.669170 7487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:17:58.669294 7487 catalog_manager.cc:1259] Loaded cluster ID: 5acafbffa24548d7bde6b3b6009abd63
I20260808 02:17:58.669329 7487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:17:58.669507 7487 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:17:58.669610 7487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Loaded TSK: 0
I20260808 02:17:58.669751 7487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:17:58.678326 6702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:17:58.679926 6702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:17:58.681074 6437 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [term 2 FOLLOWER]: Refusing update from remote peer b11f30b06888407d9ef5377a1b6cd8e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 02:17:58.681147 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.680881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35226)
0808 02:17:58.680978 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.681109 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 02:17:58.681407 7484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 02:17:58.681676 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.681554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.0.0.1:35236)
0808 02:17:58.681614 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.681647 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 02:17:58.681757 6678 consensus_peers.cc:487] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 -> Peer 50fe97b61719474d80bf4b8325421662 (127.28.177.190:42235): Couldn't send request to peer 50fe97b61719474d80bf4b8325421662. Status: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:17:58.682137 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11f30b06888407d9ef5377a1b6cd8e0. Latest consensus state: current_term: 2 leader_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:58.682263 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:17:58.682643 7463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:58.682737 7463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:17:58.683126 7490 mvcc.cc:205] Tried to move back new op lower bound from 7316092840675069952 to 7316092840624721920. Current Snapshot: MvccSnapshot[applied={T|T < 7316092840675069952}]
I20260808 02:17:58.683264 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "50fe97b61719474d80bf4b8325421662" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42235 } } peers { permanent_uuid: "f6970c77282b4a55aaf31075a30ba57f" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 38019 } attrs { promote: false } } peers { permanent_uuid: "b11f30b06888407d9ef5377a1b6cd8e0" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 33859 } attrs { promote: false } } }
I20260808 02:17:58.683336 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6970c77282b4a55aaf31075a30ba57f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:17:58.688568 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.684691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44521 (local address 127.0.0.1:35270)
0808 02:17:58.684771 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.684785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:58.685065 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:58.685336 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:58.685337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:58.685339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:58.685395 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.685396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.686276 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.686277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.687599 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.687602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.688319 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.688321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.688383 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.688390 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:17:58.688449 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:17:58.688495 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 02:17:58.688669 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.684845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.28.177.130:44521)
0808 02:17:58.685207 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:58.685210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:58.685219 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:17:58.685252 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:58.685252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:58.685265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:58.685336 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:58.685480 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.686173 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.687717 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.688180 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.688246 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.688254 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:58.688493 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 02:17:58.688546 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":91,"threads_started":1}
I20260808 02:17:58.688774 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.684905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42297 (local address 127.0.0.1:57370)
0808 02:17:58.685221 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.685232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:58.685289 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:58.685455 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:58.685456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:58.685458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:58.685493 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.685494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.686460 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.686462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.687683 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.687686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.688495 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.688497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.688559 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.688563 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:17:58.688631 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:17:58.688672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":77,"threads_started":1}
I20260808 02:17:58.688915 7056 tablet_service.cc:1467] Processing CreateTablet for tablet f9e8e4fb6ada42a6b5522c290f990de7 (DEFAULT_TABLE table=test-table [id=cc5e36ad6b5140dca9ed79ee76d043f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:58.689070 7056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e8e4fb6ada42a6b5522c290f990de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:58.689082 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.685095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.28.177.129:42297)
0808 02:17:58.685239 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:58.685241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:58.685278 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:17:58.685371 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:58.685373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:58.685384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:58.685447 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:58.685577 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.686386 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.687777 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.688366 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.688489 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.688499 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:58.688927 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:17:58.688984 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260808 02:17:58.689262 6871 tablet_service.cc:1467] Processing CreateTablet for tablet f9e8e4fb6ada42a6b5522c290f990de7 (DEFAULT_TABLE table=test-table [id=cc5e36ad6b5140dca9ed79ee76d043f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:58.689390 6871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e8e4fb6ada42a6b5522c290f990de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:58.690061 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.684995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.28.177.131:43219)
0808 02:17:58.685518 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:58.685520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:58.685526 (+ 6us) server_negotiation.cc:407] Connection header received
0808 02:17:58.685572 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:58.685573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:58.685583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:58.685665 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:17:58.685780 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.686534 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.688539 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.689685 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.689762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.689773 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:58.689929 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 02:17:58.689989 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":64,"threads_started":1}
I20260808 02:17:58.690982 7500 tablet_bootstrap.cc:492] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Bootstrap starting.
I20260808 02:17:58.691232 7501 tablet_bootstrap.cc:492] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e: Bootstrap starting.
I20260808 02:17:58.691417 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.684805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:43219 (local address 127.0.0.1:41998)
0808 02:17:58.684898 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.684908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:17:58.684964 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:17:58.685670 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:17:58.685671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:17:58.685673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:17:58.685705 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.685706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.686614 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.686615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.687715 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:17:58.687718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.689769 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.689770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:17:58.689833 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.689836 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:17:58.689921 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:17:58.691371 (+ 1450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 02:17:58.691417 7500 tablet_bootstrap.cc:654] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:58.691607 7501 tablet_bootstrap.cc:654] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:58.691905 7261 tablet_service.cc:1467] Processing CreateTablet for tablet f9e8e4fb6ada42a6b5522c290f990de7 (DEFAULT_TABLE table=test-table [id=cc5e36ad6b5140dca9ed79ee76d043f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:17:58.692031 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e8e4fb6ada42a6b5522c290f990de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:17:58.692145 7500 tablet_bootstrap.cc:492] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: No bootstrap required, opened a new log
I20260808 02:17:58.692144 7501 tablet_bootstrap.cc:492] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e: No bootstrap required, opened a new log
I20260808 02:17:58.692190 7501 ts_tablet_manager.cc:1397] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:58.692190 7500 ts_tablet_manager.cc:1397] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:58.692306 7501 raft_consensus.cc:348] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.692335 7500 raft_consensus.cc:348] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.692373 7501 raft_consensus.cc:374] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:58.692395 7501 raft_consensus.cc:729] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d8dcb828ef45e193672f5503fbf91e, State: Initialized, Role: FOLLOWER
I20260808 02:17:58.692395 7500 raft_consensus.cc:374] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:58.692416 7500 raft_consensus.cc:729] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1cb9d1f041494090fb3653a8bdb275, State: Initialized, Role: FOLLOWER
I20260808 02:17:58.692451 7501 consensus_queue.cc:260] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.692461 7500 consensus_queue.cc:260] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.692612 7500 ts_tablet_manager.cc:1428] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:58.692615 7501 ts_tablet_manager.cc:1428] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:58.693564 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.693335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.130:57941)
0808 02:17:58.693480 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.693519 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":105,"threads_started":1}
I20260808 02:17:58.693586 6964 heartbeater.cc:503] Master 127.28.177.188:33859 was elected leader, sending a full tablet report...
I20260808 02:17:58.693647 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.693405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:40707)
0808 02:17:58.693541 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.693580 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260808 02:17:58.694612 7513 tablet_bootstrap.cc:492] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Bootstrap starting.
I20260808 02:17:58.694952 7513 tablet_bootstrap.cc:654] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Neither blocks nor log segments found. Creating new log.
I20260808 02:17:58.695544 7513 tablet_bootstrap.cc:492] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: No bootstrap required, opened a new log
I20260808 02:17:58.695593 7513 ts_tablet_manager.cc:1397] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:17:58.695704 7513 raft_consensus.cc:348] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.695778 7513 raft_consensus.cc:374] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:17:58.695796 7513 raft_consensus.cc:729] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe525ab07cd4df1a2738e4f62f37db4, State: Initialized, Role: FOLLOWER
I20260808 02:17:58.695874 7513 consensus_queue.cc:260] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.696069 7513 ts_tablet_manager.cc:1428] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:17:58.696698 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.696320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.131:36719)
0808 02:17:58.696493 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.696618 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260808 02:17:58.706735 7508 raft_consensus.cc:482] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:17:58.706825 7508 raft_consensus.cc:504] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.706960 7508 leader_election.cc:288] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521), 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:58.707271 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e8e4fb6ada42a6b5522c290f990de7" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" is_pre_election: true
I20260808 02:17:58.707273 7079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e8e4fb6ada42a6b5522c290f990de7" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1cb9d1f041494090fb3653a8bdb275" is_pre_election: true
I20260808 02:17:58.707363 7290 raft_consensus.cc:2454] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 0.
I20260808 02:17:58.707386 7079 raft_consensus.cc:2454] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 0.
I20260808 02:17:58.707556 6802 leader_election.cc:302] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d8dcb828ef45e193672f5503fbf91e, 8fe525ab07cd4df1a2738e4f62f37db4; no voters:
I20260808 02:17:58.707753 7508 raft_consensus.cc:2781] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:17:58.707808 7508 raft_consensus.cc:482] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:17:58.707824 7508 raft_consensus.cc:3037] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:58.708698 7508 raft_consensus.cc:504] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.708920 7508 leader_election.cc:288] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 election: Requested vote from peers de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521), 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219)
I20260808 02:17:58.709101 7079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e8e4fb6ada42a6b5522c290f990de7" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de1cb9d1f041494090fb3653a8bdb275"
I20260808 02:17:58.709136 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e8e4fb6ada42a6b5522c290f990de7" candidate_uuid: "63d8dcb828ef45e193672f5503fbf91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe525ab07cd4df1a2738e4f62f37db4"
I20260808 02:17:58.709201 7079 raft_consensus.cc:3037] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:58.709239 7290 raft_consensus.cc:3037] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:17:58.709834 7079 raft_consensus.cc:2454] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 1.
I20260808 02:17:58.710032 7290 raft_consensus.cc:2454] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d8dcb828ef45e193672f5503fbf91e in term 1.
I20260808 02:17:58.710067 6802 leader_election.cc:302] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d8dcb828ef45e193672f5503fbf91e, de1cb9d1f041494090fb3653a8bdb275; no voters:
I20260808 02:17:58.710242 7508 raft_consensus.cc:2781] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:17:58.710427 7508 raft_consensus.cc:686] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [term 1 LEADER]: Becoming Leader. State: Replica: 63d8dcb828ef45e193672f5503fbf91e, State: Running, Role: LEADER
I20260808 02:17:58.710500 7508 consensus_queue.cc:237] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } }
I20260808 02:17:58.710929 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.710762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:46983)
0808 02:17:58.710844 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.710887 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:17:58.711472 6702 catalog_manager.cc:5495] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129). New cstate: current_term: 1 leader_uuid: "63d8dcb828ef45e193672f5503fbf91e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d8dcb828ef45e193672f5503fbf91e" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42297 } health_report { overall_health: HEALTHY } } }
I20260808 02:17:58.725706 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.723193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.28.177.129:42297)
0808 02:17:58.723266 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:17:58.723268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:17:58.723276 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:17:58.723299 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:17:58.723300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:17:58.723309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:17:58.723361 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:17:58.723504 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.724067 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.725051 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:17:58.725251 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:17:58.725323 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:17:58.725446 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:17:58.725497 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:17:58.725579 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:17:58.725620 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:17:58.725644 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:17:58.726733 7290 raft_consensus.cc:1261] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Refusing update from remote peer 63d8dcb828ef45e193672f5503fbf91e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:58.726765 7079 raft_consensus.cc:1261] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Refusing update from remote peer 63d8dcb828ef45e193672f5503fbf91e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:17:58.727010 7522 consensus_queue.cc:1035] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe525ab07cd4df1a2738e4f62f37db4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:58.727141 7508 consensus_queue.cc:1035] T f9e8e4fb6ada42a6b5522c290f990de7 P 63d8dcb828ef45e193672f5503fbf91e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de1cb9d1f041494090fb3653a8bdb275" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:17:58.727615 7153 heartbeater.cc:503] Master 127.28.177.188:33859 was elected leader, sending a full tablet report...
I20260808 02:17:58.727831 7363 heartbeater.cc:503] Master 127.28.177.188:33859 was elected leader, sending a full tablet report...
I20260808 02:17:58.728168 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.728048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.131:52347)
0808 02:17:58.728087 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.728130 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260808 02:17:58.728645 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.727985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.130:37335)
0808 02:17:58.728581 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.728615 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576}
I20260808 02:17:58.729457 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.729321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:54677)
0808 02:17:58.729396 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.729423 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:17:58.729885 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.729652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.131:51481)
0808 02:17:58.729823 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.729854 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260808 02:17:58.730756 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.730622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.130:46347)
0808 02:17:58.730696 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.730725 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 02:17:58.730819 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.730638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42235 (local address 127.28.177.129:53697)
0808 02:17:58.730756 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.730786 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 02:17:58.731417 6702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:58.731508 6702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:17:58.732875 6702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b11f30b06888407d9ef5377a1b6cd8e0: Sending DeleteTablet for 3 replicas of tablet f9e8e4fb6ada42a6b5522c290f990de7
I20260808 02:17:58.733165 7056 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e8e4fb6ada42a6b5522c290f990de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:58 UTC) from {username='oryx', principal='oryx/127.28.177.188@KRBTEST.COM'} at 127.0.0.1:35270
I20260808 02:17:58.733220 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e8e4fb6ada42a6b5522c290f990de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:58 UTC) from {username='oryx', principal='oryx/127.28.177.188@KRBTEST.COM'} at 127.0.0.1:41998
I20260808 02:17:58.733381 7540 tablet_replica.cc:330] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: stopping tablet replica
I20260808 02:17:58.733424 7541 tablet_replica.cc:330] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: stopping tablet replica
I20260808 02:17:58.733486 7541 raft_consensus.cc:2229] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:58.733515 6871 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e8e4fb6ada42a6b5522c290f990de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:17:58 UTC) from {username='oryx', principal='oryx/127.28.177.188@KRBTEST.COM'} at 127.0.0.1:57370
I20260808 02:17:58.733542 7541 raft_consensus.cc:2258] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:58.733633 7540 raft_consensus.cc:2229] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:17:58.733719 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 6648
I20260808 02:17:58.733752 7540 raft_consensus.cc:2258] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:17:58.745456 7541 ts_tablet_manager.cc:1905] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:58.746415 7540 ts_tablet_manager.cc:1905] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:17:58.747442 7541 ts_tablet_manager.cc:1918] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:58.747504 7541 log.cc:1196] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/f9e8e4fb6ada42a6b5522c290f990de7
I20260808 02:17:58.747720 7541 ts_tablet_manager.cc:1939] T f9e8e4fb6ada42a6b5522c290f990de7 P 8fe525ab07cd4df1a2738e4f62f37db4: Deleting consensus metadata
I20260808 02:17:58.748379 7540 ts_tablet_manager.cc:1918] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:17:58.748487 7540 log.cc:1196] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/f9e8e4fb6ada42a6b5522c290f990de7
I20260808 02:17:58.748458 6676 catalog_manager.cc:4841] TS 8fe525ab07cd4df1a2738e4f62f37db4 (127.28.177.131:43219): tablet f9e8e4fb6ada42a6b5522c290f990de7 (table test-table [id=cc5e36ad6b5140dca9ed79ee76d043f7]) successfully deleted
I20260808 02:17:58.748723 7540 ts_tablet_manager.cc:1939] T f9e8e4fb6ada42a6b5522c290f990de7 P de1cb9d1f041494090fb3653a8bdb275: Deleting consensus metadata
I20260808 02:17:58.749137 6676 catalog_manager.cc:4841] TS de1cb9d1f041494090fb3653a8bdb275 (127.28.177.130:44521): tablet f9e8e4fb6ada42a6b5522c290f990de7 (table test-table [id=cc5e36ad6b5140dca9ed79ee76d043f7]) successfully deleted
I20260808 02:17:58.755362 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:17:58.755254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42297 (local address 127.0.0.1:57390)
0808 02:17:58.755301 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:17:58.755327 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 02:17:58.755507 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 6973
W20260808 02:17:58.755542 6676 catalog_manager.cc:4584] TS 63d8dcb828ef45e193672f5503fbf91e (127.28.177.129:42297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9e8e4fb6ada42a6b5522c290f990de7: Network error: Client connection negotiation failed: client connection to 127.28.177.129:42297: connect: Connection refused (error 111)
I20260808 02:17:58.764285 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 7163
I20260808 02:17:58.774377 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 6240
I20260808 02:17:58.785833 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 6511
2026-08-08T02:17:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5172 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:17:58 dist-test-slave-3h5h krb5kdc[7549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:17:58 dist-test-slave-3h5h krb5kdc[7549](info): set up 2 sockets
Aug 08 02:17:58 dist-test-slave-3h5h krb5kdc[7549](info): commencing operation
krb5kdc: starting...
W20260808 02:18:00.065745 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab.
Entry for principal oryx/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab.
I20260808 02:18:00.204476 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:38889
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab
--principal=oryx/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:38889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:00.292361 7565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:00.292626 7565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:00.292670 7565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:00.293918 7565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:00.293994 7565 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:18:00.294026 7565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:00.294070 7565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:00.294101 7565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:00.294128 7565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:00.295773 7565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:38889
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.190.keytab
--principal=oryx/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:38889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:00.296078 7565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:00.296312 7565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:00.301666 7575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:00.301666 7573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:00.301666 7572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:00.302420 7565 server_base.cc:1034] running on GCE node
I20260808 02:18:00.302703 7565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:00.302977 7565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:00.304140 7565 hybrid_clock.cc:648] HybridClock initialized: now 1786155480304113 us; error 50 us; skew 500 ppm
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:00.306795 7565 init.cc:377] Logged in from keytab as oryx/127.28.177.190@KRBTEST.COM (short username oryx)
I20260808 02:18:00.307288 7565 webserver.cc:460] Webserver started at http://127.28.177.190:43049/ using document root <none> and password file <none>
I20260808 02:18:00.307544 7565 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:00.307612 7565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:00.307806 7565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:00.308983 7565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "c59267a3244549939f3c3eee751ea33e"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "b93796a3ca1f699507f4def76caf8d38"
server_key_iv: "2589f151e0e8dbfcc9ff72003abaddc7"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.309360 7565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "c59267a3244549939f3c3eee751ea33e"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "b93796a3ca1f699507f4def76caf8d38"
server_key_iv: "2589f151e0e8dbfcc9ff72003abaddc7"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.310828 7565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:18:00.311596 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.311904 7565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:00.312006 7565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "c59267a3244549939f3c3eee751ea33e"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "b93796a3ca1f699507f4def76caf8d38"
server_key_iv: "2589f151e0e8dbfcc9ff72003abaddc7"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.312084 7565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:00.351603 7565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:00.354307 7565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:00.354475 7565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:00.359754 7565 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:38889
I20260808 02:18:00.359797 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:38889 every 8 connection(s)
I20260808 02:18:00.360206 7565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:18:00.361038 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:00.361101 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 7565
I20260808 02:18:00.361202 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:18:00.361424 29382 external_mini_cluster.cc:1428] Setting key 931dbc89e03543bf2ddef4dd4685a712
I20260808 02:18:00.364439 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Bootstrap starting.
I20260808 02:18:00.365173 7671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:00.365470 7671 log.cc:824] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:00.366185 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: No bootstrap required, opened a new log
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:00.369005 7671 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59267a3244549939f3c3eee751ea33e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38889 } }
I20260808 02:18:00.369189 7671 raft_consensus.cc:374] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:00.369237 7671 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59267a3244549939f3c3eee751ea33e, State: Initialized, Role: FOLLOWER
I20260808 02:18:00.369400 7671 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59267a3244549939f3c3eee751ea33e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38889 } }
I20260808 02:18:00.369495 7671 raft_consensus.cc:388] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:18:00.369541 7671 raft_consensus.cc:482] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:18:00.369592 7671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:00.370141 7671 raft_consensus.cc:504] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59267a3244549939f3c3eee751ea33e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38889 } }
I20260808 02:18:00.370265 7671 leader_election.cc:302] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59267a3244549939f3c3eee751ea33e; no voters:
I20260808 02:18:00.370450 7671 leader_election.cc:288] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:18:00.370494 7677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:00.370661 7677 raft_consensus.cc:686] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [term 1 LEADER]: Becoming Leader. State: Replica: c59267a3244549939f3c3eee751ea33e, State: Running, Role: LEADER
I20260808 02:18:00.370738 7671 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:18:00.370770 7677 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59267a3244549939f3c3eee751ea33e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38889 } }
I20260808 02:18:00.371104 7677 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59267a3244549939f3c3eee751ea33e. Latest consensus state: current_term: 1 leader_uuid: "c59267a3244549939f3c3eee751ea33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59267a3244549939f3c3eee751ea33e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38889 } } }
I20260808 02:18:00.371194 7677 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:00.371363 7678 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c59267a3244549939f3c3eee751ea33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59267a3244549939f3c3eee751ea33e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 38889 } } }
I20260808 02:18:00.371424 7678 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:00.371558 7684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:18:00.372082 7684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:18:00.373842 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.362232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.28.177.190:38889)
0808 02:18:00.362462 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.362470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.362501 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:18:00.362946 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.362953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.363004 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.363251 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.363922 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.364959 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.365836 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.366258 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.366391 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.369388 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.369394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.369394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.369408 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.372023 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.372491 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.372494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.372494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.372605 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.372868 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.372869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.372870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.373176 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.373358 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.373470 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.373623 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20260808 02:18:00.374044 7684 catalog_manager.cc:1347] Generated new cluster ID: 7e86972135db4069af47373e96fc6920
I20260808 02:18:00.374089 7684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:18:00.406960 7684 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:18:00.407653 7684 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:18:00.412382 7684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Generated new TSK 0
I20260808 02:18:00.412622 7684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab.
Entry for principal oryx/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab.
I20260808 02:18:00.451771 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38889
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:00.539310 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:00.539548 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:00.539577 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:00.540963 7711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:00.541020 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:00.541045 7711 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:18:00.541065 7711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:18:00.542829 7711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:38889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:00.543115 7711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:00.543387 7711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:00.549245 7716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:00.549338 7711 server_base.cc:1034] running on GCE node
W20260808 02:18:00.549245 7717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:00.549311 7721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:00.549834 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:00.550130 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:00.551323 7711 hybrid_clock.cc:648] HybridClock initialized: now 1786155480551305 us; error 38 us; skew 500 ppm
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:00.554430 7711 init.cc:377] Logged in from keytab as oryx/127.28.177.129@KRBTEST.COM (short username oryx)
I20260808 02:18:00.555006 7711 webserver.cc:460] Webserver started at http://127.28.177.129:42483/ using document root <none> and password file <none>
I20260808 02:18:00.555289 7711 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:00.555343 7711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:00.555492 7711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:00.556684 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "dfc1734f169c405ba1a477a423caf002"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "f7a9b0ba2a41a37c7491d1dc776fff23"
server_key_iv: "21f33fa23aea1f778609893e4bb2af5a"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.557147 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "dfc1734f169c405ba1a477a423caf002"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "f7a9b0ba2a41a37c7491d1dc776fff23"
server_key_iv: "21f33fa23aea1f778609893e4bb2af5a"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.558655 7711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:00.559543 7730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.559754 7711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:00.559849 7711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "dfc1734f169c405ba1a477a423caf002"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "f7a9b0ba2a41a37c7491d1dc776fff23"
server_key_iv: "21f33fa23aea1f778609893e4bb2af5a"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.559902 7711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:00.578243 7711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:00.580216 7711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:00.580329 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:00.580809 7711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:00.580914 7711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.580979 7711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:00.581022 7711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:00.588416 7711 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:42253
I20260808 02:18:00.588495 7907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:42253 every 8 connection(s)
I20260808 02:18:00.588873 7711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:18:00.591749 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.581415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:44275 (local address 127.28.177.190:38889)
0808 02:18:00.581499 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.581502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.581668 (+ 166us) server_negotiation.cc:407] Connection header received
0808 02:18:00.582182 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.582183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.582199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.582286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.582999 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.583830 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.585580 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.585963 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.586043 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.589019 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.589021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.589021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.589033 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.590392 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.590808 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.590809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.590810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.590875 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.591133 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.591133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.591134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.591256 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.591317 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.591602 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.591683 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:18:00.591817 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.581372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.28.177.129:44275)
0808 02:18:00.581527 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.581569 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.582066 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.582372 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.582377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.582699 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:00.582894 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.582899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.583942 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.583944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.585453 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.585458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.586081 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.586082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.586247 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.586746 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:00.586757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:00.588916 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:00.590468 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.590475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.590477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.590711 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.590949 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.590950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.590950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.591054 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.591319 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:00.591324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:00.591442 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 02:18:00.591527 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:18:00.591658 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260808 02:18:00.592573 7910 heartbeater.cc:347] Connected to a master server at 127.28.177.190:38889
I20260808 02:18:00.592696 7910 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:00.592953 7910 heartbeater.cc:511] Master 127.28.177.190:38889 requested a full tablet report, sending...
I20260808 02:18:00.593570 7610 ts_manager.cc:194] Registered new tserver with Master: dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253)
I20260808 02:18:00.594535 7610 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:44275
I20260808 02:18:00.598387 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 7711
I20260808 02:18:00.598515 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:18:00.598744 29382 external_mini_cluster.cc:1428] Setting key dd839a90006b89565ebbfbf65d45d509
I20260808 02:18:00.600075 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.593791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:33901 (local address 127.28.177.190:38889)
0808 02:18:00.593853 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.593855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.593944 (+ 89us) server_negotiation.cc:407] Connection header received
0808 02:18:00.594029 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.594030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.594039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.594080 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.594409 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.595045 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.595851 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.596028 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.596079 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.597036 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.597037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.597038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.597044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.598670 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.599117 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.599118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.599118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.599185 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.599438 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.599438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.599439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.599617 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.599684 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.599872 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.599974 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:18:00.600075 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.593750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.28.177.129:33901)
0808 02:18:00.593879 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.593890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.593952 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.594093 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.594094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.594290 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:00.594348 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.594349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.595162 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.595163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.595779 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.595782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.596136 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.596137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.596198 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.596588 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:00.596593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:00.596958 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:00.598754 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.598755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.598756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.599034 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.599280 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.599280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.599281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.599363 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.599714 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:00.599715 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:00.599767 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:18:00.599836 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 02:18:00.599954 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab.
Entry for principal oryx/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab.
I20260808 02:18:00.631925 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38889
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:00.717181 7930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:00.717428 7930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:00.717474 7930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:00.718695 7930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:00.718766 7930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:00.718802 7930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:18:00.718845 7930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:18:00.720417 7930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:38889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:00.720712 7930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:00.720980 7930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:18:00.726747 7930 server_base.cc:1034] running on GCE node
W20260808 02:18:00.726816 7937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:00.726697 7936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:00.726707 7940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:00.727172 7930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:00.727402 7930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:00.728637 7930 hybrid_clock.cc:648] HybridClock initialized: now 1786155480728626 us; error 44 us; skew 500 ppm
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:00.731515 7930 init.cc:377] Logged in from keytab as oryx/127.28.177.130@KRBTEST.COM (short username oryx)
I20260808 02:18:00.732038 7930 webserver.cc:460] Webserver started at http://127.28.177.130:41447/ using document root <none> and password file <none>
I20260808 02:18:00.732277 7930 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:00.732352 7930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:00.732520 7930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:00.733742 7930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "72ce85083cc94a4f9a0d327f07640d6d"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "d5c4bc2d1217996b67f87f4c6c5cf962"
server_key_iv: "6c156a89b2055607f93802de28283a41"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.734170 7930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "72ce85083cc94a4f9a0d327f07640d6d"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "d5c4bc2d1217996b67f87f4c6c5cf962"
server_key_iv: "6c156a89b2055607f93802de28283a41"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.735648 7930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:18:00.736629 7952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.736948 7930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:00.737061 7930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "72ce85083cc94a4f9a0d327f07640d6d"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "d5c4bc2d1217996b67f87f4c6c5cf962"
server_key_iv: "6c156a89b2055607f93802de28283a41"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.737170 7930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:00.747890 7930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:00.750370 7930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:00.750546 7930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:00.751215 7930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:00.751271 7930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.751324 7930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:00.751377 7930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:00.759196 7930 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:46453
I20260808 02:18:00.759238 8122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:46453 every 8 connection(s)
I20260808 02:18:00.759608 7930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:18:00.761889 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.751665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:59223 (local address 127.28.177.190:38889)
0808 02:18:00.751737 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.751739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.751940 (+ 201us) server_negotiation.cc:407] Connection header received
0808 02:18:00.752423 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.752425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.752439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.752527 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.753261 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.753806 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.754861 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.755118 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.755199 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.758695 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.758698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.758699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.758706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.760331 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.760791 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.760792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.760793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.761002 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.761270 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.761271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.761271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.761388 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.761448 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.761750 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.761824 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:18:00.762034 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.751655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.28.177.130:59223)
0808 02:18:00.751802 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.751844 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.752327 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.752579 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.752582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.752826 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:00.753159 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.753165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.753909 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.753910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.754749 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.754751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.755208 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.755209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.755389 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.755954 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:00.755965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:00.758556 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:00.760438 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.760444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.760445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.760692 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.761076 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.761077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.761078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.761202 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.761460 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:00.761465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:00.761607 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 02:18:00.761698 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:18:00.761825 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260808 02:18:00.762648 8123 heartbeater.cc:347] Connected to a master server at 127.28.177.190:38889
I20260808 02:18:00.762784 8123 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:00.762974 8123 heartbeater.cc:511] Master 127.28.177.190:38889 requested a full tablet report, sending...
I20260808 02:18:00.763410 7609 ts_manager.cc:194] Registered new tserver with Master: 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453)
I20260808 02:18:00.763942 7609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:59223
I20260808 02:18:00.768328 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 7930
I20260808 02:18:00.768433 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:18:00.768720 29382 external_mini_cluster.cc:1428] Setting key ffee9607383db3414dd255664676d348
I20260808 02:18:00.769513 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.763833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:54899 (local address 127.28.177.190:38889)
0808 02:18:00.763874 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.763876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.764099 (+ 223us) server_negotiation.cc:407] Connection header received
0808 02:18:00.764129 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.764130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.764141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.764221 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.764523 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.765066 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.765787 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.765988 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.766074 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.766919 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.766921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.766921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.766928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.768131 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.768505 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.768507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.768507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.768578 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.768793 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.768794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.768795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.769025 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.769178 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.769340 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.769420 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:18:00.769527 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.763732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.28.177.130:54899)
0808 02:18:00.763918 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.763929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.763988 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.764225 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.764226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.764384 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:00.764439 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.764440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.765156 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.765157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.765691 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.765693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.766077 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.766078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.766152 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.766547 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:00.766551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:00.766817 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:00.768215 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.768216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.768216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.768417 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.768656 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.768657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.768657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.768720 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.769169 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:00.769170 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:00.769223 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 02:18:00.769312 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:18:00.769420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for oryx/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab.
Entry for principal oryx/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab.
I20260808 02:18:00.800714 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:38889
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:00.892638 8137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:00.892910 8137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:00.892940 8137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:00.894167 8137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:00.894209 8137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:00.894229 8137 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:18:00.894248 8137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:18:00.895887 8137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.177.190:38889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:00.896147 8137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:00.896389 8137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:00.901648 8145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:00.901753 8137 server_base.cc:1034] running on GCE node
W20260808 02:18:00.901670 8143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:00.901648 8142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:00.902070 8137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:00.902362 8137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:00.903529 8137 hybrid_clock.cc:648] HybridClock initialized: now 1786155480903507 us; error 40 us; skew 500 ppm
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:00.906304 8137 init.cc:377] Logged in from keytab as oryx/127.28.177.131@KRBTEST.COM (short username oryx)
I20260808 02:18:00.906787 8137 webserver.cc:460] Webserver started at http://127.28.177.131:37841/ using document root <none> and password file <none>
I20260808 02:18:00.907025 8137 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:00.907100 8137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:00.907244 8137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:00.908304 8137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "106eb3c024e04db2a04126bb7064da14"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "4176b9ad96fe869bfe48c6cb2b0bc743"
server_key_iv: "9a50728f87b687c76e89cb1f95de3b66"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.908718 8137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "106eb3c024e04db2a04126bb7064da14"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "4176b9ad96fe869bfe48c6cb2b0bc743"
server_key_iv: "9a50728f87b687c76e89cb1f95de3b66"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.910140 8137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:00.910853 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.911026 8137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.911134 8137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "106eb3c024e04db2a04126bb7064da14"
format_stamp: "Formatted at 2026-08-08 02:18:00 on dist-test-slave-3h5h"
server_key: "4176b9ad96fe869bfe48c6cb2b0bc743"
server_key_iv: "9a50728f87b687c76e89cb1f95de3b66"
server_key_version: "encryptionkey@0"
I20260808 02:18:00.911204 8137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:00.921146 8137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:00.923956 8137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:00.924157 8137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:00.924921 8137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:00.924996 8137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.925028 8137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:00.925057 8137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:00.935652 8137 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:35581
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:00.935986 8329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:35581 every 8 connection(s)
I20260808 02:18:00.936089 8137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:18:00.937016 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 8137
I20260808 02:18:00.937119 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:18:00.937343 29382 external_mini_cluster.cc:1428] Setting key 6b5c9387bcd4acb1d462ece10121ed69
I20260808 02:18:00.942417 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.925262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:60447 (local address 127.28.177.190:38889)
0808 02:18:00.925314 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.925317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.925874 (+ 557us) server_negotiation.cc:407] Connection header received
0808 02:18:00.926601 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.926602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.926616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.926675 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.927739 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.928628 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.933018 (+ 4390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.933373 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.933465 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.936429 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.936431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.936431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.936442 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.940547 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.941151 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.941153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.941154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.941226 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.941560 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.941561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.941561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.941737 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.941819 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.942221 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.942319 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:18:00.942643 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.925438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.28.177.131:60447)
0808 02:18:00.925709 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.925755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.926501 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.926803 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.926808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.927260 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:00.927558 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.927564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.928724 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.928726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.930193 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.930197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.933505 (+ 3308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.933506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.933705 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.934282 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:00.934295 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:00.936324 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:00.940658 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.940666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.940668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.941039 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.941320 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.941321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.941322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.941493 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.941828 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:00.941834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:00.942035 (+ 201us) client_negotiation.cc:769] Sending connection context
0808 02:18:00.942170 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 02:18:00.942369 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260808 02:18:00.946000 8330 heartbeater.cc:347] Connected to a master server at 127.28.177.190:38889
I20260808 02:18:00.946071 8330 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:00.946291 8330 heartbeater.cc:511] Master 127.28.177.190:38889 requested a full tablet report, sending...
I20260808 02:18:00.946704 7610 ts_manager.cc:194] Registered new tserver with Master: 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:00.947747 7610 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:60447
I20260808 02:18:00.949870 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:18:00.953624 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.946175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:42185 (local address 127.28.177.190:38889)
0808 02:18:00.946244 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.946246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.946462 (+ 216us) server_negotiation.cc:407] Connection header received
0808 02:18:00.946566 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.946568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.946580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.946618 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.947106 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.947754 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.948765 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.949136 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.949210 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.950471 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.950473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.950474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.950483 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.952334 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.952687 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.952689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.952689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.952759 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.953013 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.953013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.953014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.953163 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.953242 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.953422 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.953512 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:18:00.953646 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.946237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.28.177.131:42185)
0808 02:18:00.946390 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.946402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.946489 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.946688 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.946688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.946913 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:00.946992 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.946994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.947858 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.947859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.948669 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.948672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.949270 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.949272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:00.949357 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.949961 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:00.949967 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:00.950365 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:00.952429 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.952430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.952430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.952613 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.952857 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:00.952857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:00.952858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:00.952934 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:00.953232 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:00.953233 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:00.953291 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:18:00.953379 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:18:00.953510 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:18:00.982504 7675 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:00.974767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.28.177.190:38889)
0808 02:18:00.974815 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.974817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.974858 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:18:00.974955 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.974956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.974967 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.975020 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.975958 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.976464 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.977752 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.977944 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.977996 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.982282 (+ 4286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34538: BlockingRecv error: recv got EOF from 127.0.0.1:34538 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 08 02:18:00 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:00.991430 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.983320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.28.177.190:38889)
0808 02:18:00.983364 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.983367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.983501 (+ 134us) server_negotiation.cc:407] Connection header received
0808 02:18:00.983537 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.983538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.983553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.983635 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:00.984075 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.984894 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.985671 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.985900 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:00.985993 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:00.988481 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:00.988483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:00.988483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:00.988492 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:00.990117 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.990519 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.990520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.990521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.990596 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:00.990828 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:00.990829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:00.990829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:00.990980 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:00.991112 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:00.991222 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 02:18:00.991309 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:18:00.993793 7610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:18:00.995572 7610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:18:01.002490 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.998195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42253 (local address 127.0.0.1:34054)
0808 02:18:00.998712 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.998729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.998885 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.999138 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.999139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.999142 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:00.999199 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.999201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.000101 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.000102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.001414 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.001417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.002166 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.002167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.002249 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.002263 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.002340 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.002393 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1,"thread_start_us":48,"threads_started":1}
I20260808 02:18:01.002496 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.998541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.28.177.129:42253)
0808 02:18:00.998734 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.998739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.998858 (+ 119us) server_negotiation.cc:407] Connection header received
0808 02:18:00.998964 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.998967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.998983 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.999056 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:00.999280 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:00.999973 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.001503 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.002059 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.002182 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.002247 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.002353 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.002393 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260808 02:18:01.002867 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.998385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:35581 (local address 127.0.0.1:42102)
0808 02:18:00.998786 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.998795 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.998855 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.999204 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.999205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.999206 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:00.999260 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.999261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.000481 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.000482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.001541 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.001543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.002618 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.002620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.002703 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.002707 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.002755 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.002791 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":66,"threads_started":1}
I20260808 02:18:01.003115 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.998490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.28.177.131:35581)
0808 02:18:00.998670 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.998674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.998995 (+ 321us) server_negotiation.cc:407] Connection header received
0808 02:18:00.999097 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.999102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.999121 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.999195 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:00.999324 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.000383 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.001633 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.002412 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.002508 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.002560 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.002983 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.003025 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":123,"threads_started":1}
I20260808 02:18:01.003367 7800 tablet_service.cc:1467] Processing CreateTablet for tablet 682c92a45d97452ebd54fb6f744405a5 (DEFAULT_TABLE table=test-table [id=8a407618802f42538b07190edca2c30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.003826 7800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682c92a45d97452ebd54fb6f744405a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.003809 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 682c92a45d97452ebd54fb6f744405a5 (DEFAULT_TABLE table=test-table [id=8a407618802f42538b07190edca2c30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.004161 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682c92a45d97452ebd54fb6f744405a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.005160 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.998188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:46453 (local address 127.0.0.1:34670)
0808 02:18:00.998544 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:00.998584 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:00.998666 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:00.999017 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:00.999020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:00.999022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:00.999100 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:00.999102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.000453 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.000454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.001553 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.001561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.004924 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.004925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.004990 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.004994 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.005047 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.005090 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":91,"threads_started":1}
I20260808 02:18:01.005553 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:00.998323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.28.177.130:46453)
0808 02:18:00.998523 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:00.998528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:00.998770 (+ 242us) server_negotiation.cc:407] Connection header received
0808 02:18:00.998809 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:00.998811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:00.998824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:00.998915 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:00.999189 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.000356 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.001668 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.002412 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.002509 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.005356 (+ 2847us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.005385 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.005435 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":122,"threads_started":1}
I20260808 02:18:01.006240 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 682c92a45d97452ebd54fb6f744405a5 (DEFAULT_TABLE table=test-table [id=8a407618802f42538b07190edca2c30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.006652 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682c92a45d97452ebd54fb6f744405a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.007375 8374 tablet_bootstrap.cc:492] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Bootstrap starting.
I20260808 02:18:01.008296 8374 tablet_bootstrap.cc:654] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.008675 8374 log.cc:824] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:01.009001 8375 tablet_bootstrap.cc:492] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Bootstrap starting.
I20260808 02:18:01.009619 8374 tablet_bootstrap.cc:492] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: No bootstrap required, opened a new log
I20260808 02:18:01.009702 8374 ts_tablet_manager.cc:1397] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:18:01.009735 8375 tablet_bootstrap.cc:654] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.010053 8375 log.cc:824] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:01.011051 8375 tablet_bootstrap.cc:492] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: No bootstrap required, opened a new log
I20260808 02:18:01.011122 8375 ts_tablet_manager.cc:1397] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:18:01.011992 8374 raft_consensus.cc:348] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.012166 8374 raft_consensus.cc:374] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.012204 8374 raft_consensus.cc:729] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106eb3c024e04db2a04126bb7064da14, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.012323 8374 consensus_queue.cc:260] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.012601 8374 ts_tablet_manager.cc:1428] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:18:01.012625 8330 heartbeater.cc:503] Master 127.28.177.190:38889 was elected leader, sending a full tablet report...
I20260808 02:18:01.012804 8377 tablet_bootstrap.cc:492] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Bootstrap starting.
I20260808 02:18:01.013253 8375 raft_consensus.cc:348] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.013412 8375 raft_consensus.cc:374] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.013453 8375 raft_consensus.cc:729] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.013490 8377 tablet_bootstrap.cc:654] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.013602 8375 consensus_queue.cc:260] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.013833 8377 log.cc:824] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:01.013906 8375 ts_tablet_manager.cc:1428] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:18:01.014086 7910 heartbeater.cc:503] Master 127.28.177.190:38889 was elected leader, sending a full tablet report...
I20260808 02:18:01.014636 8377 tablet_bootstrap.cc:492] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: No bootstrap required, opened a new log
I20260808 02:18:01.014693 8377 ts_tablet_manager.cc:1397] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:18:01.016393 8377 raft_consensus.cc:348] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.016525 8377 raft_consensus.cc:374] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.016546 8377 raft_consensus.cc:729] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.016685 8377 consensus_queue.cc:260] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.017004 8123 heartbeater.cc:503] Master 127.28.177.190:38889 was elected leader, sending a full tablet report...
I20260808 02:18:01.017001 8377 ts_tablet_manager.cc:1428] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 02:18:01.026211 8384 raft_consensus.cc:482] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:18:01.026321 8384 raft_consensus.cc:504] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.026623 8384 leader_election.cc:288] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.030769 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.026956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:36319 (local address 127.28.177.129:42253)
0808 02:18:01.027007 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.027009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.027018 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:01.027050 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.027051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.027062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.027123 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:01.027312 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.028022 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.029776 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.030335 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.030411 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.030420 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.030657 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.030698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:18:01.030769 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.027230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:42153 (local address 127.28.177.131:35581)
0808 02:18:01.027271 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.027273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.027282 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:01.027357 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.027358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.027367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.027422 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:01.027683 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.028288 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.029771 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.030329 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.030411 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.030420 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.030657 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.030698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:18:01.030836 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.027001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:35581 (local address 127.28.177.130:42153)
0808 02:18:01.027224 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.027236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.027285 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.027502 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.027503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.027505 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:01.027591 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.027593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.028377 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.028378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.029662 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.029664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.030436 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.030438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.030559 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.030562 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.030633 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.030680 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":5,"thread_start_us":116,"threads_started":1}
I20260808 02:18:01.030851 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.026878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42253 (local address 127.28.177.130:36319)
0808 02:18:01.026935 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.026949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.026991 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.027155 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.027157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.027159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:01.027238 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.027240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.028142 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.028143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.029657 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.029660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.030436 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.030438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.030554 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.030559 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.030633 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.030680 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":7}
I20260808 02:18:01.031304 7836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682c92a45d97452ebd54fb6f744405a5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc1734f169c405ba1a477a423caf002" is_pre_election: true
I20260808 02:18:01.031313 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682c92a45d97452ebd54fb6f744405a5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14" is_pre_election: true
I20260808 02:18:01.031430 7836 raft_consensus.cc:2454] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 0.
I20260808 02:18:01.031487 8267 raft_consensus.cc:2454] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 0.
I20260808 02:18:01.031745 7956 leader_election.cc:302] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, 72ce85083cc94a4f9a0d327f07640d6d; no voters:
I20260808 02:18:01.031906 8384 raft_consensus.cc:2781] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:18:01.032007 8384 raft_consensus.cc:482] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:18:01.032040 8384 raft_consensus.cc:3037] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.033241 8384 raft_consensus.cc:504] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.033404 8384 leader_election.cc:288] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 election: Requested vote from peers dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.033604 7836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682c92a45d97452ebd54fb6f744405a5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc1734f169c405ba1a477a423caf002"
I20260808 02:18:01.033663 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682c92a45d97452ebd54fb6f744405a5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14"
I20260808 02:18:01.033718 7836 raft_consensus.cc:3037] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.033785 8267 raft_consensus.cc:3037] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.034524 8267 raft_consensus.cc:2454] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 1.
I20260808 02:18:01.034590 7836 raft_consensus.cc:2454] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 1.
I20260808 02:18:01.034741 7956 leader_election.cc:302] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, 72ce85083cc94a4f9a0d327f07640d6d; no voters:
I20260808 02:18:01.034905 8384 raft_consensus.cc:2781] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:01.035073 8384 raft_consensus.cc:686] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 LEADER]: Becoming Leader. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Running, Role: LEADER
I20260808 02:18:01.035216 8384 consensus_queue.cc:237] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.036123 7610 catalog_manager.cc:5495] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130). New cstate: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } health_report { overall_health: UNKNOWN } } }
I20260808 02:18:01.049535 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.047037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.28.177.130:46453)
0808 02:18:01.047114 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.047116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.047128 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:18:01.047179 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.047180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.047198 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.047259 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.047390 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.047880 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.048712 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.048969 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.049029 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.049216 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.049318 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.049380 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.049420 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.049447 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:18:01.051730 8267 raft_consensus.cc:1261] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Refusing update from remote peer 72ce85083cc94a4f9a0d327f07640d6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.051741 7836 raft_consensus.cc:1261] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Refusing update from remote peer 72ce85083cc94a4f9a0d327f07640d6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.052141 8384 consensus_queue.cc:1035] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.052316 8387 consensus_queue.cc:1035] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.055629 8392 mvcc.cc:205] Tried to move back new op lower bound from 7316092850383966208 to 7316092850320883712. Current Snapshot: MvccSnapshot[applied={T|T < 7316092850383966208}]
I20260808 02:18:01.055651 8394 mvcc.cc:205] Tried to move back new op lower bound from 7316092850383966208 to 7316092850320883712. Current Snapshot: MvccSnapshot[applied={T|T < 7316092850383966208}]
I20260808 02:18:01.056264 8391 mvcc.cc:205] Tried to move back new op lower bound from 7316092850383966208 to 7316092850320883712. Current Snapshot: MvccSnapshot[applied={T|T < 7316092850383966208 or (T in {7316092850383966208})}]
I20260808 02:18:01.058018 7609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.058212 7609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.059159 7609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Sending DeleteTablet for 3 replicas of tablet 682c92a45d97452ebd54fb6f744405a5
I20260808 02:18:01.059536 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 682c92a45d97452ebd54fb6f744405a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34670
I20260808 02:18:01.059556 7800 tablet_service.cc:1514] Processing DeleteTablet for tablet 682c92a45d97452ebd54fb6f744405a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34054
I20260808 02:18:01.059693 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet 682c92a45d97452ebd54fb6f744405a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42102
I20260808 02:18:01.059782 8407 tablet_replica.cc:330] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: stopping tablet replica
I20260808 02:18:01.059796 8406 tablet_replica.cc:330] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: stopping tablet replica
I20260808 02:18:01.059873 8407 raft_consensus.cc:2229] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:18:01.059921 8406 raft_consensus.cc:2229] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.059964 8407 raft_consensus.cc:2258] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.060042 8406 raft_consensus.cc:2258] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.060361 7609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:18:01.060540 8407 ts_tablet_manager.cc:1905] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 02:18:01.060549 7609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:18:01.060804 8408 tablet_replica.cc:330] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: stopping tablet replica
I20260808 02:18:01.061206 8408 raft_consensus.cc:2229] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.061407 8408 raft_consensus.cc:2258] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.062278 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 924db3ccaab54267b0d05307f13194f7 (DEFAULT_TABLE table=test-table [id=0fc10df4ceb14431a2cb5f4ae2a7a688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.062328 7800 tablet_service.cc:1467] Processing CreateTablet for tablet 924db3ccaab54267b0d05307f13194f7 (DEFAULT_TABLE table=test-table [id=0fc10df4ceb14431a2cb5f4ae2a7a688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.062444 7800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924db3ccaab54267b0d05307f13194f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.062469 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924db3ccaab54267b0d05307f13194f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.062500 8407 ts_tablet_manager.cc:1918] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:01.062572 8407 log.cc:1196] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/682c92a45d97452ebd54fb6f744405a5
I20260808 02:18:01.062848 8407 ts_tablet_manager.cc:1939] T 682c92a45d97452ebd54fb6f744405a5 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting consensus metadata
I20260808 02:18:01.063284 7587 catalog_manager.cc:4841] TS 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453): tablet 682c92a45d97452ebd54fb6f744405a5 (table test-table [id=8a407618802f42538b07190edca2c30c]) successfully deleted
I20260808 02:18:01.063911 8377 tablet_bootstrap.cc:492] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Bootstrap starting.
I20260808 02:18:01.064357 8377 tablet_bootstrap.cc:654] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.064392 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 924db3ccaab54267b0d05307f13194f7 (DEFAULT_TABLE table=test-table [id=0fc10df4ceb14431a2cb5f4ae2a7a688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.064510 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924db3ccaab54267b0d05307f13194f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.064555 8406 ts_tablet_manager.cc:1905] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.065119 8377 tablet_bootstrap.cc:492] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: No bootstrap required, opened a new log
I20260808 02:18:01.065161 8377 ts_tablet_manager.cc:1397] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.065304 8377 raft_consensus.cc:348] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.065367 8377 raft_consensus.cc:374] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.065388 8377 raft_consensus.cc:729] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.065433 8377 consensus_queue.cc:260] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.065515 8377 ts_tablet_manager.cc:1428] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:18:01.066021 8374 tablet_bootstrap.cc:492] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Bootstrap starting.
I20260808 02:18:01.066198 8375 tablet_bootstrap.cc:492] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Bootstrap starting.
I20260808 02:18:01.066524 8374 tablet_bootstrap.cc:654] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.066672 8375 tablet_bootstrap.cc:654] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.067209 8374 tablet_bootstrap.cc:492] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: No bootstrap required, opened a new log
I20260808 02:18:01.067252 8374 ts_tablet_manager.cc:1397] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.067396 8374 raft_consensus.cc:348] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.067469 8374 raft_consensus.cc:374] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.067492 8374 raft_consensus.cc:729] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106eb3c024e04db2a04126bb7064da14, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.067534 8374 consensus_queue.cc:260] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.067617 8374 ts_tablet_manager.cc:1428] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:18:01.061998 8408 ts_tablet_manager.cc:1905] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.067682 8406 ts_tablet_manager.cc:1918] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:01.067778 8406 log.cc:1196] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/682c92a45d97452ebd54fb6f744405a5
I20260808 02:18:01.068130 8406 ts_tablet_manager.cc:1939] T 682c92a45d97452ebd54fb6f744405a5 P dfc1734f169c405ba1a477a423caf002: Deleting consensus metadata
I20260808 02:18:01.068619 7588 catalog_manager.cc:4841] TS dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253): tablet 682c92a45d97452ebd54fb6f744405a5 (table test-table [id=8a407618802f42538b07190edca2c30c]) successfully deleted
I20260808 02:18:01.069295 8375 tablet_bootstrap.cc:492] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: No bootstrap required, opened a new log
I20260808 02:18:01.069353 8375 ts_tablet_manager.cc:1397] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:18:01.069505 8375 raft_consensus.cc:348] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.069578 8375 raft_consensus.cc:374] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.069598 8375 raft_consensus.cc:729] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.069563 8408 ts_tablet_manager.cc:1918] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:01.069646 8375 consensus_queue.cc:260] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.069681 8408 log.cc:1196] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/682c92a45d97452ebd54fb6f744405a5
I20260808 02:18:01.069738 8375 ts_tablet_manager.cc:1428] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 02:18:01.069980 8408 ts_tablet_manager.cc:1939] T 682c92a45d97452ebd54fb6f744405a5 P 106eb3c024e04db2a04126bb7064da14: Deleting consensus metadata
I20260808 02:18:01.070444 7588 catalog_manager.cc:4841] TS 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581): tablet 682c92a45d97452ebd54fb6f744405a5 (table test-table [id=8a407618802f42538b07190edca2c30c]) successfully deleted
W20260808 02:18:01.089785 7911 tablet.cc:2369] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:18:01.124179 8382 raft_consensus.cc:482] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:18:01.124284 8382 raft_consensus.cc:504] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.124703 8382 leader_election.cc:288] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.128686 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.125080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:35581 (local address 127.28.177.129:45157)
0808 02:18:01.125230 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.125239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.125306 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.125617 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.125618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.125621 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:01.125678 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.125680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.126464 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.126465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.127495 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.127498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.128385 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.128387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.128479 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.128484 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.128551 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.128596 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":57,"thread_start_us":74,"threads_started":1}
I20260808 02:18:01.128692 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.125193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:45157 (local address 127.28.177.131:35581)
0808 02:18:01.125241 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.125243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.125292 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:18:01.125521 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.125522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.125532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.125586 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:01.125764 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.126356 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.127652 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.128269 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.128355 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.128366 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.128558 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.128611 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:18:01.128794 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.124994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:58793 (local address 127.28.177.130:46453)
0808 02:18:01.125052 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.125055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.125263 (+ 208us) server_negotiation.cc:407] Connection header received
0808 02:18:01.125374 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.125375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.125386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.125455 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:01.125678 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.126378 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.127756 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.128382 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.128457 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.128467 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.128661 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.128704 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:18:01.129041 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.124894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:46453 (local address 127.28.177.129:58793)
0808 02:18:01.125181 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.125193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.125298 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.125477 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.125478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.125480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:01.125543 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.125546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.126495 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.126496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.127653 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.127657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.128478 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.128479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.128562 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.128567 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.128929 (+ 362us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.128982 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":153,"threads_started":1}
I20260808 02:18:01.129432 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924db3ccaab54267b0d05307f13194f7" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14" is_pre_election: true
I20260808 02:18:01.129551 8267 raft_consensus.cc:2454] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 0.
I20260808 02:18:01.129739 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924db3ccaab54267b0d05307f13194f7" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ce85083cc94a4f9a0d327f07640d6d" is_pre_election: true
I20260808 02:18:01.129772 7733 leader_election.cc:302] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.129895 8059 raft_consensus.cc:2454] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 0.
I20260808 02:18:01.129925 8382 raft_consensus.cc:2781] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:18:01.129961 8382 raft_consensus.cc:482] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:18:01.129987 8382 raft_consensus.cc:3037] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.130910 8382 raft_consensus.cc:504] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.131069 8382 leader_election.cc:288] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 election: Requested vote from peers 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.131273 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924db3ccaab54267b0d05307f13194f7" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14"
I20260808 02:18:01.131372 8267 raft_consensus.cc:3037] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.132066 8267 raft_consensus.cc:2454] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 1.
I20260808 02:18:01.132229 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924db3ccaab54267b0d05307f13194f7" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ce85083cc94a4f9a0d327f07640d6d"
I20260808 02:18:01.132294 7733 leader_election.cc:302] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.132339 8059 raft_consensus.cc:3037] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.132423 8382 raft_consensus.cc:2781] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:01.132555 8382 raft_consensus.cc:686] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 1 LEADER]: Becoming Leader. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Running, Role: LEADER
I20260808 02:18:01.132632 8382 consensus_queue.cc:237] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.133358 8059 raft_consensus.cc:2454] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 1.
I20260808 02:18:01.133428 7610 catalog_manager.cc:5495] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfc1734f169c405ba1a477a423caf002 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "dfc1734f169c405ba1a477a423caf002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155480, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.190@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:01.178211 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.169746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:38889 (local address 127.0.0.1:34558)
0808 02:18:01.169945 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.169957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.170028 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.170214 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.170215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.170571 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:01.170674 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.170677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.171730 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.171731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.172500 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.172504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.173033 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.173035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.173092 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.173737 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:01.173742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:01.175260 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:01.177093 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:01.177101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:01.177103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:01.177297 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:01.177529 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:01.177530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:01.177530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:01.177598 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:01.177906 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:01.177914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:01.177961 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.178021 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.178124 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260808 02:18:01.178310 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.169845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.28.177.190:38889)
0808 02:18:01.169921 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.169923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.170033 (+ 110us) server_negotiation.cc:407] Connection header received
0808 02:18:01.170086 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.170087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.170101 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.170185 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:01.170804 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.171600 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.172606 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.172942 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.173015 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.175370 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:01.175372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:01.175373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:01.175383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:01.176994 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:01.177371 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:01.177372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:01.177373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:01.177434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:01.177706 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:01.177707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:01.177708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:01.177836 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:01.177902 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.178068 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.178188 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":44}
I20260808 02:18:01.180536 7610 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34558:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:18:01.180764 7610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:18:01.182421 7800 tablet_service.cc:1467] Processing CreateTablet for tablet 471540e9bec044f9b938349ea1c8faa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d14dd57b10f14988bdb57f64ba278cc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:18:01.182432 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 471540e9bec044f9b938349ea1c8faa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d14dd57b10f14988bdb57f64ba278cc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:18:01.182600 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471540e9bec044f9b938349ea1c8faa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.182600 7800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471540e9bec044f9b938349ea1c8faa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.184204 8377 tablet_bootstrap.cc:492] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: Bootstrap starting.
I20260808 02:18:01.184230 8375 tablet_bootstrap.cc:492] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: Bootstrap starting.
I20260808 02:18:01.184303 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 471540e9bec044f9b938349ea1c8faa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d14dd57b10f14988bdb57f64ba278cc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:18:01.184455 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471540e9bec044f9b938349ea1c8faa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.184543 8377 tablet_bootstrap.cc:654] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.184680 8375 tablet_bootstrap.cc:654] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.185452 8375 tablet_bootstrap.cc:492] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: No bootstrap required, opened a new log
I20260808 02:18:01.185509 8375 ts_tablet_manager.cc:1397] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.185667 8375 raft_consensus.cc:348] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.185758 8375 raft_consensus.cc:374] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.185781 8375 raft_consensus.cc:729] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.185832 8375 consensus_queue.cc:260] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.185938 8375 ts_tablet_manager.cc:1428] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:18:01.186058 8382 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.186124 8374 tablet_bootstrap.cc:492] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: Bootstrap starting.
I20260808 02:18:01.186137 8382 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.186342 8377 tablet_bootstrap.cc:492] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: No bootstrap required, opened a new log
I20260808 02:18:01.186393 8377 ts_tablet_manager.cc:1397] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:18:01.186497 8377 raft_consensus.cc:348] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.186555 8374 tablet_bootstrap.cc:654] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.186581 8377 raft_consensus.cc:374] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.186606 8377 raft_consensus.cc:729] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.186663 8377 consensus_queue.cc:260] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.186770 8377 ts_tablet_manager.cc:1428] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:01.186836 8398 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.186954 8398 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.187315 8374 tablet_bootstrap.cc:492] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: No bootstrap required, opened a new log
I20260808 02:18:01.187372 8374 ts_tablet_manager.cc:1397] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.187525 8374 raft_consensus.cc:348] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.187626 8374 raft_consensus.cc:374] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.187666 8374 raft_consensus.cc:729] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106eb3c024e04db2a04126bb7064da14, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.187716 8374 consensus_queue.cc:260] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.187825 8374 ts_tablet_manager.cc:1428] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260808 02:18:01.187862 8331 tablet.cc:2369] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:18:01.187889 8381 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.187963 8381 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.195906 8398 raft_consensus.cc:482] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:18:01.196004 8398 raft_consensus.cc:504] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.196261 8398 leader_election.cc:288] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.196462 7836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471540e9bec044f9b938349ea1c8faa5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc1734f169c405ba1a477a423caf002" is_pre_election: true
I20260808 02:18:01.196475 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471540e9bec044f9b938349ea1c8faa5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14" is_pre_election: true
I20260808 02:18:01.196571 8267 raft_consensus.cc:2454] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 0.
I20260808 02:18:01.196576 7836 raft_consensus.cc:2454] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 0.
I20260808 02:18:01.196731 7956 leader_election.cc:302] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ce85083cc94a4f9a0d327f07640d6d, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.196883 8398 raft_consensus.cc:2781] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:18:01.196926 8398 raft_consensus.cc:482] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:18:01.196951 8398 raft_consensus.cc:3037] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.197588 8398 raft_consensus.cc:504] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.197715 8398 leader_election.cc:288] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 election: Requested vote from peers dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.197885 7836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471540e9bec044f9b938349ea1c8faa5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc1734f169c405ba1a477a423caf002"
I20260808 02:18:01.197894 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471540e9bec044f9b938349ea1c8faa5" candidate_uuid: "72ce85083cc94a4f9a0d327f07640d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14"
I20260808 02:18:01.197962 7836 raft_consensus.cc:3037] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.197980 8267 raft_consensus.cc:3037] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.198572 7836 raft_consensus.cc:2454] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 1.
I20260808 02:18:01.198663 8267 raft_consensus.cc:2454] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ce85083cc94a4f9a0d327f07640d6d in term 1.
I20260808 02:18:01.198737 7956 leader_election.cc:302] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ce85083cc94a4f9a0d327f07640d6d, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.198827 8398 raft_consensus.cc:2781] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:01.198880 8398 raft_consensus.cc:686] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 LEADER]: Becoming Leader. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Running, Role: LEADER
I20260808 02:18:01.198931 8398 consensus_queue.cc:237] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.199260 8387 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: TxnStatusTablet state changed. Reason: New leader 72ce85083cc94a4f9a0d327f07640d6d. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.199360 8387 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:18:01.199575 8434 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:18:01.199705 8434 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:18:01.199643 7609 catalog_manager.cc:5495] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130). New cstate: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } health_report { overall_health: UNKNOWN } } }
I20260808 02:18:01.205675 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.202723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.28.177.130:46453)
0808 02:18:01.202765 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.202768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.202803 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:18:01.202894 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.202895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.202903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.202946 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.203222 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.203739 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.204873 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.205161 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.205226 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.205442 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.205473 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.205513 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.205599 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.205619 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:18:01.205727 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.202668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:46453 (local address 127.0.0.1:34692)
0808 02:18:01.202758 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.202769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.202823 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.202963 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.202965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.202966 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:18:01.203038 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.203040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.203837 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.203839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.204743 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.204746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.205248 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.205250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.205322 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.205345 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.205535 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.205542 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.205596 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.205646 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:18:01.206750 7836 raft_consensus.cc:1261] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Refusing update from remote peer 72ce85083cc94a4f9a0d327f07640d6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.206748 8267 raft_consensus.cc:1261] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Refusing update from remote peer 72ce85083cc94a4f9a0d327f07640d6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.207103 8387 consensus_queue.cc:1035] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.207376 8387 consensus_queue.cc:1035] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.207788 8381 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: TxnStatusTablet state changed. Reason: New leader 72ce85083cc94a4f9a0d327f07640d6d. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.207914 8381 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.208317 8387 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.208407 8387 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:18:01.209141 8382 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: TxnStatusTablet state changed. Reason: New leader 72ce85083cc94a4f9a0d327f07640d6d. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.209237 8382 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.209520 8384 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.209602 8384 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 72ce85083cc94a4f9a0d327f07640d6d: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:18:01.210196 8382 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.210276 8382 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P dfc1734f169c405ba1a477a423caf002: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.211035 8381 tablet_replica.cc:439] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72ce85083cc94a4f9a0d327f07640d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } } }
I20260808 02:18:01.211125 8381 tablet_replica.cc:442] T 471540e9bec044f9b938349ea1c8faa5 P 106eb3c024e04db2a04126bb7064da14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:18:01.214182 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.210974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.28.177.129:42253)
0808 02:18:01.211027 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.211029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.211178 (+ 149us) server_negotiation.cc:407] Connection header received
0808 02:18:01.211271 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.211272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.211294 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.211371 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.211581 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.212132 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.213264 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.213449 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.213510 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.213753 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.213869 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.213953 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.214031 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.214061 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:18:01.220391 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.217724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53209 (local address 127.28.177.130:46453)
0808 02:18:01.217772 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.217774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.217916 (+ 142us) server_negotiation.cc:407] Connection header received
0808 02:18:01.218093 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.218094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.218104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.218175 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.218327 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.218818 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.219644 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.219850 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.219937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.220130 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.220163 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.220235 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.220295 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.220316 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:18:01.220418 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.217608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:46453 (local address 127.28.177.129:53209)
0808 02:18:01.217840 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.217850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.217934 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.218180 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.218181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.218183 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:18:01.218247 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.218249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.218901 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.218902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.219543 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.219545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.219943 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.219944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.220007 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.220030 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.220235 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.220238 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.220291 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.220335 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260808 02:18:01.223018 8059 raft_consensus.cc:1261] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Refusing update from remote peer dfc1734f169c405ba1a477a423caf002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.223207 8267 raft_consensus.cc:1261] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Refusing update from remote peer dfc1734f169c405ba1a477a423caf002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.223316 8382 consensus_queue.cc:1035] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.223448 8419 consensus_queue.cc:1035] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.232502 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.229961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:44657 (local address 127.28.177.129:42253)
0808 02:18:01.230018 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.230020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.230029 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:01.230073 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.230075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.230085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.230129 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.230254 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.230731 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.231808 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.231996 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.232074 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.232247 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.232286 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.232348 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.232409 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.232430 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:18:01.232527 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.229892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42253 (local address 127.28.177.130:44657)
0808 02:18:01.229957 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.229969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.230017 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.230140 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.230141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.230142 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:18:01.230187 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.230188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.230815 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.230816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.231711 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.231713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.232073 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.232074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.232130 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.232145 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.232348 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.232351 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.232404 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.232449 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 02:18:01.242080 7609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.242204 7609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.242913 7609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Sending DeleteTablet for 3 replicas of tablet 924db3ccaab54267b0d05307f13194f7
I20260808 02:18:01.243175 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 924db3ccaab54267b0d05307f13194f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34670
I20260808 02:18:01.243228 7800 tablet_service.cc:1514] Processing DeleteTablet for tablet 924db3ccaab54267b0d05307f13194f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34054
I20260808 02:18:01.243273 8407 tablet_replica.cc:330] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: stopping tablet replica
I20260808 02:18:01.243316 8406 tablet_replica.cc:330] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: stopping tablet replica
I20260808 02:18:01.243322 8407 raft_consensus.cc:2229] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.243361 8406 raft_consensus.cc:2229] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:18:01.243358 8407 raft_consensus.cc:2258] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.243470 8406 raft_consensus.cc:2258] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.243884 8406 ts_tablet_manager.cc:1905] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.243897 8407 ts_tablet_manager.cc:1905] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.245674 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet 924db3ccaab54267b0d05307f13194f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42102
I20260808 02:18:01.245744 8406 ts_tablet_manager.cc:1918] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:18:01.245782 8408 tablet_replica.cc:330] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: stopping tablet replica
I20260808 02:18:01.245801 8406 log.cc:1196] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/924db3ccaab54267b0d05307f13194f7
I20260808 02:18:01.245843 8408 raft_consensus.cc:2229] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.245885 8408 raft_consensus.cc:2258] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.246044 8406 ts_tablet_manager.cc:1939] T 924db3ccaab54267b0d05307f13194f7 P dfc1734f169c405ba1a477a423caf002: Deleting consensus metadata
I20260808 02:18:01.246227 8408 ts_tablet_manager.cc:1905] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.246483 8407 ts_tablet_manager.cc:1918] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:18:01.246543 8407 log.cc:1196] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/924db3ccaab54267b0d05307f13194f7
I20260808 02:18:01.246758 8407 ts_tablet_manager.cc:1939] T 924db3ccaab54267b0d05307f13194f7 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting consensus metadata
I20260808 02:18:01.247092 7587 catalog_manager.cc:4841] TS 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453): tablet 924db3ccaab54267b0d05307f13194f7 (table test-table [id=0fc10df4ceb14431a2cb5f4ae2a7a688]) successfully deleted
I20260808 02:18:01.248088 8408 ts_tablet_manager.cc:1918] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:18:01.248167 8408 log.cc:1196] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/924db3ccaab54267b0d05307f13194f7
I20260808 02:18:01.248399 8408 ts_tablet_manager.cc:1939] T 924db3ccaab54267b0d05307f13194f7 P 106eb3c024e04db2a04126bb7064da14: Deleting consensus metadata
I20260808 02:18:01.249140 7588 catalog_manager.cc:4841] TS dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253): tablet 924db3ccaab54267b0d05307f13194f7 (table test-table [id=0fc10df4ceb14431a2cb5f4ae2a7a688]) successfully deleted
I20260808 02:18:01.249351 7588 catalog_manager.cc:4841] TS 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581): tablet 924db3ccaab54267b0d05307f13194f7 (table test-table [id=0fc10df4ceb14431a2cb5f4ae2a7a688]) successfully deleted
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[7549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:18:01.279327 7675 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:01.272013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.28.177.190:38889)
0808 02:18:01.272066 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.272069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.272299 (+ 230us) server_negotiation.cc:407] Connection header received
0808 02:18:01.272435 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.272437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.272455 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.272526 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:01.273323 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.274145 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.275265 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.275579 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.275658 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.279253 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34568: BlockingRecv error: recv got EOF from 127.0.0.1:34568 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[7549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM
I20260808 02:18:01.289638 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.280587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.28.177.190:38889)
0808 02:18:01.280630 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.280632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.280894 (+ 262us) server_negotiation.cc:407] Connection header received
0808 02:18:01.280957 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.280958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.280970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.281030 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:01.281636 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.282367 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.283403 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.283658 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.283733 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.286594 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:01.286596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:01.286596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:01.286605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:01.288342 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:01.288765 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:01.288767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:01.288767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:01.288864 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:01.289073 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:01.289074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:01.289074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:01.289214 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:01.289312 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.289449 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.289532 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:18:01.291620 7609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:18:01.291801 7609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:18:01.293453 8032 tablet_service.cc:1467] Processing CreateTablet for tablet f7c4564347694b47aa0595612bc3336b (DEFAULT_TABLE table=test-table [id=95805e38899846138090d50c2a9c9d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.293506 7800 tablet_service.cc:1467] Processing CreateTablet for tablet f7c4564347694b47aa0595612bc3336b (DEFAULT_TABLE table=test-table [id=95805e38899846138090d50c2a9c9d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.293596 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c4564347694b47aa0595612bc3336b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.293660 7800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c4564347694b47aa0595612bc3336b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.294961 8377 tablet_bootstrap.cc:492] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Bootstrap starting.
I20260808 02:18:01.295147 8233 tablet_service.cc:1467] Processing CreateTablet for tablet f7c4564347694b47aa0595612bc3336b (DEFAULT_TABLE table=test-table [id=95805e38899846138090d50c2a9c9d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.295274 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c4564347694b47aa0595612bc3336b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.295287 8377 tablet_bootstrap.cc:654] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.296658 8375 tablet_bootstrap.cc:492] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Bootstrap starting.
I20260808 02:18:01.296626 8374 tablet_bootstrap.cc:492] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Bootstrap starting.
I20260808 02:18:01.297288 8375 tablet_bootstrap.cc:654] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.297930 8374 tablet_bootstrap.cc:654] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.298010 8377 tablet_bootstrap.cc:492] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: No bootstrap required, opened a new log
I20260808 02:18:01.298056 8377 ts_tablet_manager.cc:1397] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 02:18:01.298170 8377 raft_consensus.cc:348] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.298238 8377 raft_consensus.cc:374] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.298259 8377 raft_consensus.cc:729] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.298309 8377 consensus_queue.cc:260] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.298401 8377 ts_tablet_manager.cc:1428] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:01.299445 8374 tablet_bootstrap.cc:492] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: No bootstrap required, opened a new log
I20260808 02:18:01.299499 8374 ts_tablet_manager.cc:1397] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 02:18:01.299682 8374 raft_consensus.cc:348] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.299767 8374 raft_consensus.cc:374] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.299796 8374 raft_consensus.cc:729] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106eb3c024e04db2a04126bb7064da14, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.299837 8374 consensus_queue.cc:260] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.299945 8374 ts_tablet_manager.cc:1428] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:18:01.305545 8375 tablet_bootstrap.cc:492] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: No bootstrap required, opened a new log
I20260808 02:18:01.305682 8375 ts_tablet_manager.cc:1397] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260808 02:18:01.305907 8375 raft_consensus.cc:348] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.306128 8375 raft_consensus.cc:374] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.306164 8375 raft_consensus.cc:729] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.306263 8375 consensus_queue.cc:260] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.306408 8375 ts_tablet_manager.cc:1428] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260808 02:18:01.510689 8125 tablet.cc:2369] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:18:01.568468 8419 raft_consensus.cc:482] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:18:01.568635 8419 raft_consensus.cc:504] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.568878 8419 leader_election.cc:288] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.569154 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c4564347694b47aa0595612bc3336b" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14" is_pre_election: true
I20260808 02:18:01.569262 8267 raft_consensus.cc:2454] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 0.
I20260808 02:18:01.569310 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c4564347694b47aa0595612bc3336b" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ce85083cc94a4f9a0d327f07640d6d" is_pre_election: true
I20260808 02:18:01.569458 8059 raft_consensus.cc:2454] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 0.
I20260808 02:18:01.569449 7733 leader_election.cc:302] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.569572 8419 raft_consensus.cc:2781] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:18:01.569630 8419 raft_consensus.cc:482] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:18:01.569674 8419 raft_consensus.cc:3037] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.570561 8419 raft_consensus.cc:504] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.570770 8419 leader_election.cc:288] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 election: Requested vote from peers 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453), 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581)
I20260808 02:18:01.570967 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c4564347694b47aa0595612bc3336b" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106eb3c024e04db2a04126bb7064da14"
I20260808 02:18:01.570969 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c4564347694b47aa0595612bc3336b" candidate_uuid: "dfc1734f169c405ba1a477a423caf002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ce85083cc94a4f9a0d327f07640d6d"
I20260808 02:18:01.571071 8267 raft_consensus.cc:3037] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.571071 8059 raft_consensus.cc:3037] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.571998 8267 raft_consensus.cc:2454] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 1.
I20260808 02:18:01.572002 8059 raft_consensus.cc:2454] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc1734f169c405ba1a477a423caf002 in term 1.
I20260808 02:18:01.572224 7733 leader_election.cc:302] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.572340 8419 raft_consensus.cc:2781] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:01.572397 8419 raft_consensus.cc:686] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 1 LEADER]: Becoming Leader. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Running, Role: LEADER
I20260808 02:18:01.572443 8419 consensus_queue.cc:237] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.573283 7609 catalog_manager.cc:5495] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfc1734f169c405ba1a477a423caf002 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "dfc1734f169c405ba1a477a423caf002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } health_report { overall_health: UNKNOWN } } }
I20260808 02:18:01.586513 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.583738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.28.177.129:42253)
0808 02:18:01.583820 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.583822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.583856 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:18:01.583883 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.583884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.583899 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.583952 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.584219 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.584763 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.585698 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.585906 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.585982 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.586174 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.586207 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.586255 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.586321 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.586345 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:18:01.587558 8267 raft_consensus.cc:1261] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Refusing update from remote peer dfc1734f169c405ba1a477a423caf002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.587574 8059 raft_consensus.cc:1261] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Refusing update from remote peer dfc1734f169c405ba1a477a423caf002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.587829 8382 consensus_queue.cc:1035] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.587996 8419 consensus_queue.cc:1035] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.590914 7610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.591041 7610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.591743 7610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Sending DeleteTablet for 3 replicas of tablet f7c4564347694b47aa0595612bc3336b
I20260808 02:18:01.591996 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c4564347694b47aa0595612bc3336b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42102
I20260808 02:18:01.592099 8408 tablet_replica.cc:330] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: stopping tablet replica
I20260808 02:18:01.592070 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c4564347694b47aa0595612bc3336b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34670
I20260808 02:18:01.592164 8408 raft_consensus.cc:2229] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.592150 7800 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c4564347694b47aa0595612bc3336b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34054
I20260808 02:18:01.592181 8407 tablet_replica.cc:330] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: stopping tablet replica
I20260808 02:18:01.592216 8408 raft_consensus.cc:2258] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.592233 8407 raft_consensus.cc:2229] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.592296 8407 raft_consensus.cc:2258] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.592387 8406 tablet_replica.cc:330] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: stopping tablet replica
I20260808 02:18:01.592442 8406 raft_consensus.cc:2229] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:18:01.592530 8406 raft_consensus.cc:2258] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.592761 7602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:18:01.593017 7602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:18:01.593245 8407 ts_tablet_manager.cc:1905] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.593329 8408 ts_tablet_manager.cc:1905] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.593427 8406 ts_tablet_manager.cc:1905] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.594358 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 144850189d0143d9b2872338fd48aa35 (DEFAULT_TABLE table=test-table [id=f3dd1cb6279e4f29bf6b12a9cd0e788e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.594461 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144850189d0143d9b2872338fd48aa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.595266 8233 tablet_service.cc:1467] Processing CreateTablet for tablet 144850189d0143d9b2872338fd48aa35 (DEFAULT_TABLE table=test-table [id=f3dd1cb6279e4f29bf6b12a9cd0e788e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.595335 7800 tablet_service.cc:1467] Processing CreateTablet for tablet 144850189d0143d9b2872338fd48aa35 (DEFAULT_TABLE table=test-table [id=f3dd1cb6279e4f29bf6b12a9cd0e788e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:01.595450 8407 ts_tablet_manager.cc:1918] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:01.595489 8406 ts_tablet_manager.cc:1918] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:01.595500 8408 ts_tablet_manager.cc:1918] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:01.595532 8407 log.cc:1196] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/f7c4564347694b47aa0595612bc3336b
I20260808 02:18:01.595536 8406 log.cc:1196] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/f7c4564347694b47aa0595612bc3336b
I20260808 02:18:01.595558 8408 log.cc:1196] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/f7c4564347694b47aa0595612bc3336b
I20260808 02:18:01.595803 8406 ts_tablet_manager.cc:1939] T f7c4564347694b47aa0595612bc3336b P dfc1734f169c405ba1a477a423caf002: Deleting consensus metadata
I20260808 02:18:01.595819 8408 ts_tablet_manager.cc:1939] T f7c4564347694b47aa0595612bc3336b P 106eb3c024e04db2a04126bb7064da14: Deleting consensus metadata
I20260808 02:18:01.595837 8407 ts_tablet_manager.cc:1939] T f7c4564347694b47aa0595612bc3336b P 72ce85083cc94a4f9a0d327f07640d6d: Deleting consensus metadata
I20260808 02:18:01.595947 7800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144850189d0143d9b2872338fd48aa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.595975 8233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144850189d0143d9b2872338fd48aa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:01.596247 8377 tablet_bootstrap.cc:492] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Bootstrap starting.
I20260808 02:18:01.596598 8377 tablet_bootstrap.cc:654] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.596908 8375 tablet_bootstrap.cc:492] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Bootstrap starting.
I20260808 02:18:01.597078 7587 catalog_manager.cc:4841] TS 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453): tablet f7c4564347694b47aa0595612bc3336b (table test-table [id=95805e38899846138090d50c2a9c9d78]) successfully deleted
I20260808 02:18:01.597224 8375 tablet_bootstrap.cc:654] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.597302 8377 tablet_bootstrap.cc:492] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: No bootstrap required, opened a new log
I20260808 02:18:01.597352 8377 ts_tablet_manager.cc:1397] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.597461 8377 raft_consensus.cc:348] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.597487 7588 catalog_manager.cc:4841] TS dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253): tablet f7c4564347694b47aa0595612bc3336b (table test-table [id=95805e38899846138090d50c2a9c9d78]) successfully deleted
I20260808 02:18:01.597532 8377 raft_consensus.cc:374] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.597558 8377 raft_consensus.cc:729] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ce85083cc94a4f9a0d327f07640d6d, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.597581 8374 tablet_bootstrap.cc:492] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Bootstrap starting.
I20260808 02:18:01.597607 8377 consensus_queue.cc:260] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.597724 8377 ts_tablet_manager.cc:1428] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 02:18:01.598016 8374 tablet_bootstrap.cc:654] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:01.598287 7588 catalog_manager.cc:4841] TS 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581): tablet f7c4564347694b47aa0595612bc3336b (table test-table [id=95805e38899846138090d50c2a9c9d78]) successfully deleted
I20260808 02:18:01.598446 8375 tablet_bootstrap.cc:492] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: No bootstrap required, opened a new log
I20260808 02:18:01.598502 8375 ts_tablet_manager.cc:1397] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260808 02:18:01.598615 8375 raft_consensus.cc:348] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.598670 8375 raft_consensus.cc:374] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.598683 8375 raft_consensus.cc:729] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc1734f169c405ba1a477a423caf002, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.598721 8374 tablet_bootstrap.cc:492] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: No bootstrap required, opened a new log
I20260808 02:18:01.598740 8375 consensus_queue.cc:260] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.598770 8374 ts_tablet_manager.cc:1397] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.598842 8375 ts_tablet_manager.cc:1428] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:01.598922 8374 raft_consensus.cc:348] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.598999 8374 raft_consensus.cc:374] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:01.599020 8374 raft_consensus.cc:729] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106eb3c024e04db2a04126bb7064da14, State: Initialized, Role: FOLLOWER
I20260808 02:18:01.599081 8374 consensus_queue.cc:260] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.599514 8374 ts_tablet_manager.cc:1428] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:01.746842 8381 raft_consensus.cc:482] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:18:01.746949 8381 raft_consensus.cc:504] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.747428 8381 leader_election.cc:288] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253), 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453)
I20260808 02:18:01.752141 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.747694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42253 (local address 127.28.177.131:34931)
0808 02:18:01.747856 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.747870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.747947 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.748151 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.748152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.748158 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:01.748289 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.748290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.749343 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.749345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.750918 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.750921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.751740 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.751741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.751858 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.751864 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.751945 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.751999 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":130,"thread_start_us":65,"threads_started":1}
I20260808 02:18:01.752187 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.747760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:34931 (local address 127.28.177.129:42253)
0808 02:18:01.747811 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.747813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.747930 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:18:01.748028 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.748029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.748040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.748118 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:01.748428 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.749228 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.751038 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.751639 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.751722 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.751732 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.751973 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.752013 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:18:01.752322 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.747694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:46453 (local address 127.28.177.131:47335)
0808 02:18:01.747855 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.747870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.747947 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.748151 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.748152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.748156 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:01.748256 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.748259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.749366 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.749367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.750917 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.750921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.751849 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.751851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.751916 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.751920 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.752101 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.752149 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":57,"thread_start_us":81,"threads_started":1}
I20260808 02:18:01.752353 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.747820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:47335 (local address 127.28.177.130:46453)
0808 02:18:01.747928 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.747930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.747941 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:18:01.748028 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.748029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.748040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.748122 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:01.748429 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.749246 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.751155 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.751740 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.751807 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.751814 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.752211 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.752266 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
I20260808 02:18:01.752650 7836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144850189d0143d9b2872338fd48aa35" candidate_uuid: "106eb3c024e04db2a04126bb7064da14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc1734f169c405ba1a477a423caf002" is_pre_election: true
I20260808 02:18:01.752760 7836 raft_consensus.cc:2454] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106eb3c024e04db2a04126bb7064da14 in term 0.
I20260808 02:18:01.752769 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144850189d0143d9b2872338fd48aa35" candidate_uuid: "106eb3c024e04db2a04126bb7064da14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ce85083cc94a4f9a0d327f07640d6d" is_pre_election: true
I20260808 02:18:01.752936 8059 raft_consensus.cc:2454] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106eb3c024e04db2a04126bb7064da14 in term 0.
I20260808 02:18:01.752966 8161 leader_election.cc:302] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.753116 8381 raft_consensus.cc:2781] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:18:01.753162 8381 raft_consensus.cc:482] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:18:01.753182 8381 raft_consensus.cc:3037] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.753913 8381 raft_consensus.cc:504] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.754082 8381 leader_election.cc:288] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [CANDIDATE]: Term 1 election: Requested vote from peers dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253), 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453)
I20260808 02:18:01.754251 7836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144850189d0143d9b2872338fd48aa35" candidate_uuid: "106eb3c024e04db2a04126bb7064da14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc1734f169c405ba1a477a423caf002"
I20260808 02:18:01.754317 7836 raft_consensus.cc:3037] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.754642 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144850189d0143d9b2872338fd48aa35" candidate_uuid: "106eb3c024e04db2a04126bb7064da14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ce85083cc94a4f9a0d327f07640d6d"
I20260808 02:18:01.754740 8059 raft_consensus.cc:3037] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:01.755041 7836 raft_consensus.cc:2454] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106eb3c024e04db2a04126bb7064da14 in term 1.
I20260808 02:18:01.755272 8161 leader_election.cc:302] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106eb3c024e04db2a04126bb7064da14, dfc1734f169c405ba1a477a423caf002; no voters:
I20260808 02:18:01.755364 8381 raft_consensus.cc:2781] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:01.755492 8381 raft_consensus.cc:686] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 1 LEADER]: Becoming Leader. State: Replica: 106eb3c024e04db2a04126bb7064da14, State: Running, Role: LEADER
I20260808 02:18:01.755573 8381 consensus_queue.cc:237] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } }
I20260808 02:18:01.755704 8059 raft_consensus.cc:2454] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106eb3c024e04db2a04126bb7064da14 in term 1.
I20260808 02:18:01.756341 7602 catalog_manager.cc:5495] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 106eb3c024e04db2a04126bb7064da14 (127.28.177.131). New cstate: current_term: 1 leader_uuid: "106eb3c024e04db2a04126bb7064da14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106eb3c024e04db2a04126bb7064da14" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 35581 } health_report { overall_health: HEALTHY } } }
I20260808 02:18:01.787699 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.784739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.28.177.131:35581)
0808 02:18:01.784968 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.784970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.784978 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:18:01.785027 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.785028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.785058 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.785156 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.785347 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.785934 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.786801 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.787023 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.787101 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.787262 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.787377 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.787443 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.787541 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.787560 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":148,"threads_started":1}
I20260808 02:18:01.793774 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.790892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:55085 (local address 127.28.177.130:46453)
0808 02:18:01.790947 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.790949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.791011 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:18:01.791038 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.791039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.791050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.791116 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.791317 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.791838 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.792879 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.793183 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.793278 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.793492 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.793532 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.793586 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.793646 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.793672 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:18:01.793813 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.790602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:46453 (local address 127.28.177.131:55085)
0808 02:18:01.790868 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.790882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.790954 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.791165 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.791166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.791168 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:18:01.791241 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.791242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.791972 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.791973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.792758 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.792761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.793281 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.793283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.793364 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.793378 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.793588 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.793591 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.793643 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.793706 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260808 02:18:01.796370 8059 raft_consensus.cc:1261] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Refusing update from remote peer 106eb3c024e04db2a04126bb7064da14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.796370 7836 raft_consensus.cc:1261] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Refusing update from remote peer 106eb3c024e04db2a04126bb7064da14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:01.796650 8381 consensus_queue.cc:1035] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc1734f169c405ba1a477a423caf002" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 42253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.796795 8492 consensus_queue.cc:1035] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72ce85083cc94a4f9a0d327f07640d6d" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:01.805384 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.802693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:50985 (local address 127.28.177.131:35581)
0808 02:18:01.802741 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:01.802743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:01.802841 (+ 98us) server_negotiation.cc:407] Connection header received
0808 02:18:01.802900 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:01.802901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:01.802910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:01.802971 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:01.803060 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.803572 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.804597 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.804939 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.805093 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.805213 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.805241 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.805280 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:01.805313 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 02:18:01.805330 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:18:01.805413 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.802632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:35581 (local address 127.28.177.130:50985)
0808 02:18:01.802788 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.802799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:01.802859 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:01.802977 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:01.802978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:01.802979 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:18:01.803022 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.803023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.803635 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.803636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.804480 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:01.804483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:01.805100 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:01.805101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:01.805146 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:01.805151 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:18:01.805280 (+ 129us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:18:01.805281 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 02:18:01.805311 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 02:18:01.805346 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
I20260808 02:18:01.815497 7610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.815653 7610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:01.816390 7610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59267a3244549939f3c3eee751ea33e: Sending DeleteTablet for 3 replicas of tablet 144850189d0143d9b2872338fd48aa35
I20260808 02:18:01.816679 7800 tablet_service.cc:1514] Processing DeleteTablet for tablet 144850189d0143d9b2872338fd48aa35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34054
I20260808 02:18:01.816681 8233 tablet_service.cc:1514] Processing DeleteTablet for tablet 144850189d0143d9b2872338fd48aa35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42102
I20260808 02:18:01.816788 8406 tablet_replica.cc:330] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: stopping tablet replica
I20260808 02:18:01.816788 8408 tablet_replica.cc:330] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: stopping tablet replica
I20260808 02:18:01.816874 8408 raft_consensus.cc:2229] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:18:01.816876 8406 raft_consensus.cc:2229] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.816921 8406 raft_consensus.cc:2258] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.816977 8408 raft_consensus.cc:2258] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.817365 8408 ts_tablet_manager.cc:1905] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.817368 8406 ts_tablet_manager.cc:1905] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.819129 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 144850189d0143d9b2872338fd48aa35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:01 UTC) from {username='oryx', principal='oryx/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34670
I20260808 02:18:01.819257 8407 tablet_replica.cc:330] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: stopping tablet replica
I20260808 02:18:01.819249 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 7711
I20260808 02:18:01.819320 8407 raft_consensus.cc:2229] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:01.819372 8407 raft_consensus.cc:2258] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:01.819448 8406 ts_tablet_manager.cc:1918] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:18:01.819522 8406 log.cc:1196] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/144850189d0143d9b2872338fd48aa35
I20260808 02:18:01.819794 8406 ts_tablet_manager.cc:1939] T 144850189d0143d9b2872338fd48aa35 P dfc1734f169c405ba1a477a423caf002: Deleting consensus metadata
I20260808 02:18:01.820288 8407 ts_tablet_manager.cc:1905] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:01.822449 8407 ts_tablet_manager.cc:1918] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:18:01.822535 8407 log.cc:1196] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/144850189d0143d9b2872338fd48aa35
I20260808 02:18:01.822795 8407 ts_tablet_manager.cc:1939] T 144850189d0143d9b2872338fd48aa35 P 72ce85083cc94a4f9a0d327f07640d6d: Deleting consensus metadata
I20260808 02:18:01.824812 7587 catalog_manager.cc:4841] TS 72ce85083cc94a4f9a0d327f07640d6d (127.28.177.130:46453): tablet 144850189d0143d9b2872338fd48aa35 (table test-table [id=f3dd1cb6279e4f29bf6b12a9cd0e788e]) successfully deleted
I20260808 02:18:01.829288 8408 ts_tablet_manager.cc:1918] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:18:01.829376 8408 log.cc:1196] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/144850189d0143d9b2872338fd48aa35
I20260808 02:18:01.829639 8408 ts_tablet_manager.cc:1939] T 144850189d0143d9b2872338fd48aa35 P 106eb3c024e04db2a04126bb7064da14: Deleting consensus metadata
I20260808 02:18:01.830031 7588 catalog_manager.cc:4841] TS 106eb3c024e04db2a04126bb7064da14 (127.28.177.131:35581): tablet 144850189d0143d9b2872338fd48aa35 (table test-table [id=f3dd1cb6279e4f29bf6b12a9cd0e788e]) successfully deleted
W20260808 02:18:01.833066 7588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:42253 (error 108)
I20260808 02:18:01.833304 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 7930
I20260808 02:18:01.843681 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:01.843408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:42253 (local address 127.0.0.1:34078)
0808 02:18:01.843559 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:01.843630 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:42253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
W20260808 02:18:01.843922 7588 catalog_manager.cc:4584] TS dfc1734f169c405ba1a477a423caf002 (127.28.177.129:42253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 144850189d0143d9b2872338fd48aa35: Network error: Client connection negotiation failed: client connection to 127.28.177.129:42253: connect: Connection refused (error 111)
I20260808 02:18:01.851404 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 8137
I20260808 02:18:01.867848 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 7565
2026-08-08T02:18:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3084 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[8526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[8526](info): set up 2 sockets
Aug 08 02:18:01 dist-test-slave-3h5h krb5kdc[8526](info): commencing operation
krb5kdc: starting...
W20260808 02:18:03.166263 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:18:03 dist-test-slave-3h5h krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:18:03.296700 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:33479
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:33479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:03.383651 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:03.383909 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:03.383934 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:03.385164 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:03.385207 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:03.385221 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:03.385471 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:03.385499 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:03.387434 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:33479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:03.387737 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:03.388017 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:03.393483 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:03.393483 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:03.393497 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:03.393857 8542 server_base.cc:1034] running on GCE node
I20260808 02:18:03.394186 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:03.394459 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:03.395646 8542 hybrid_clock.cc:648] HybridClock initialized: now 1786155483395627 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T02:18:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1529 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:18:03 dist-test-slave-3h5h krb5kdc[8560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:18:03 dist-test-slave-3h5h krb5kdc[8560](info): set up 2 sockets
Aug 08 02:18:03 dist-test-slave-3h5h krb5kdc[8560](info): commencing operation
krb5kdc: starting...
W20260808 02:18:04.639782 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:18:04 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:18:04.776042 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42207
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:04.862287 8582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:04.862550 8582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:04.862592 8582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:04.863798 8582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:04.863875 8582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:04.863898 8582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:04.863917 8582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:04.863948 8582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:04.865545 8582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:04.865860 8582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:04.866103 8582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:04.871675 8593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:04.871678 8591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:04.871681 8589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:04.871843 8582 server_base.cc:1034] running on GCE node
I20260808 02:18:04.872169 8582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:04.872478 8582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:04.873623 8582 hybrid_clock.cc:648] HybridClock initialized: now 1786155484873608 us; error 33 us; skew 500 ppm
Aug 08 02:18:04 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155484, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:04.883239 8582 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:18:04.883879 8582 webserver.cc:460] Webserver started at http://127.28.177.190:46101/ using document root <none> and password file <none>
I20260808 02:18:04.884305 8582 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:04.884392 8582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:04.884567 8582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:04.885742 8582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "50678c902f61437caa599b402619958b"
format_stamp: "Formatted at 2026-08-08 02:18:04 on dist-test-slave-3h5h"
server_key: "6c57200ec1ec0024b2792b8d3c36960c"
server_key_iv: "30dbae696c45986b0a6fbada66302938"
server_key_version: "encryptionkey@0"
I20260808 02:18:04.886129 8582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "50678c902f61437caa599b402619958b"
format_stamp: "Formatted at 2026-08-08 02:18:04 on dist-test-slave-3h5h"
server_key: "6c57200ec1ec0024b2792b8d3c36960c"
server_key_iv: "30dbae696c45986b0a6fbada66302938"
server_key_version: "encryptionkey@0"
I20260808 02:18:04.887605 8582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:18:04.888441 8603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:04.888652 8582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:04.888772 8582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "50678c902f61437caa599b402619958b"
format_stamp: "Formatted at 2026-08-08 02:18:04 on dist-test-slave-3h5h"
server_key: "6c57200ec1ec0024b2792b8d3c36960c"
server_key_iv: "30dbae696c45986b0a6fbada66302938"
server_key_version: "encryptionkey@0"
I20260808 02:18:04.888870 8582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:04.905313 8582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:04.907487 8582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:04.907663 8582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:04.912002 8582 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42207
I20260808 02:18:04.912140 8681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42207 every 8 connection(s)
I20260808 02:18:04.912461 8582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:18:04.913508 8683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:04.915791 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Bootstrap starting.
I20260808 02:18:04.916364 8683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:04.916623 8683 log.cc:824] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:04.917368 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: No bootstrap required, opened a new log
I20260808 02:18:04.919039 8683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:04.919190 8683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:04.919214 8683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50678c902f61437caa599b402619958b, State: Initialized, Role: FOLLOWER
I20260808 02:18:04.919322 8683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:04.919404 8683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:18:04.919445 8683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:18:04.919521 8683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:04.920097 8683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:04.920226 8683 leader_election.cc:302] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50678c902f61437caa599b402619958b; no voters:
I20260808 02:18:04.920416 8683 leader_election.cc:288] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:18:04.920476 8688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:04.920696 8683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:18:04.920998 8688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 LEADER]: Becoming Leader. State: Replica: 50678c902f61437caa599b402619958b, State: Running, Role: LEADER
I20260808 02:18:04.921106 8688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:04.921553 8689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50678c902f61437caa599b402619958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } } }
I20260808 02:18:04.921656 8689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:04.921554 8690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50678c902f61437caa599b402619958b. Latest consensus state: current_term: 1 leader_uuid: "50678c902f61437caa599b402619958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } } }
I20260808 02:18:04.921700 8690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:04.921860 8702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:18:04.922399 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 8582
I20260808 02:18:04.922408 8702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:18:04.922521 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:18:04.922772 29382 external_mini_cluster.cc:1428] Setting key 467d0a24ebc62a0e985301a7161cbc26
I20260808 02:18:04.924227 8702 catalog_manager.cc:1347] Generated new cluster ID: 94714c5ec5404ab6a55e05e4a7319f55
I20260808 02:18:04.924291 8702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:18:04 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:04.935174 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:04.923258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.28.177.190:42207)
0808 02:18:04.923416 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:04.923428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:04.923523 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:18:04.923896 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:04.923904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:04.923961 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:04.924971 (+ 1010us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:04.925081 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:04.926610 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:04.927690 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:04.928360 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:04.928480 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:04.931457 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:04.931461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:04.931462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:04.931477 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:04.933454 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:04.933898 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:04.933902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:04.933902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:04.934009 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:04.934231 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:04.934231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:04.934232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:04.934538 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:04.934687 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:04.934809 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 02:18:04.934941 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260808 02:18:04.948704 8702 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:18:04.949425 8702 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:18:04.962415 8702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Generated new TSK 0
I20260808 02:18:04.962625 8702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:18:04.996657 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42207
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:05.087610 8722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:05.087817 8722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:05.087844 8722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:05.089069 8722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:05.089112 8722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:05.089139 8722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:18:05.090607 8722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:05.090868 8722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:05.091116 8722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:05.096544 8733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.096551 8731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.096578 8730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:05.096714 8722 server_base.cc:1034] running on GCE node
I20260808 02:18:05.097074 8722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:05.097297 8722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:05.098450 8722 hybrid_clock.cc:648] HybridClock initialized: now 1786155485098433 us; error 43 us; skew 500 ppm
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:05.101197 8722 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:18:05.101634 8722 webserver.cc:460] Webserver started at http://127.28.177.129:42295/ using document root <none> and password file <none>
I20260808 02:18:05.101884 8722 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:05.101953 8722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:05.102113 8722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:05.103284 8722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "c5477fe34a9b486d97613e494f8967c1"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "9bd8877661841355ba8c9842cc0fdaad"
server_key_iv: "738b823d11287a79bc362de123bbc17f"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.103673 8722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "c5477fe34a9b486d97613e494f8967c1"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "9bd8877661841355ba8c9842cc0fdaad"
server_key_iv: "738b823d11287a79bc362de123bbc17f"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.105218 8722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:18:05.106032 8742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.106258 8722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:05.106355 8722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "c5477fe34a9b486d97613e494f8967c1"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "9bd8877661841355ba8c9842cc0fdaad"
server_key_iv: "738b823d11287a79bc362de123bbc17f"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.106443 8722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:05.120378 8722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:05.123030 8722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:05.123208 8722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:05.123468 8722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:05.123798 8722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:05.123853 8722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.123890 8722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:05.123917 8722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.130096 8722 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:44005
I20260808 02:18:05.130155 8923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:44005 every 8 connection(s)
I20260808 02:18:05.130520 8722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:18:05.132916 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 8722
I20260808 02:18:05.133023 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:18:05.133227 29382 external_mini_cluster.cc:1428] Setting key b1f2ad5c4bae397f90a6b268e625f087
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:05.141050 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.131294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:49415 (local address 127.28.177.190:42207)
0808 02:18:05.131374 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:05.131376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:05.131661 (+ 285us) server_negotiation.cc:407] Connection header received
0808 02:18:05.132168 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:05.132169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:05.132179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:05.132241 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:05.133437 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.134012 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.135016 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.135350 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.135428 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.138031 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:05.138032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:05.138033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:05.138049 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:05.139554 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:05.140000 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:05.140002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:05.140002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:05.140065 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:05.140342 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:05.140343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:05.140343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:05.140464 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:05.140539 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:05.140858 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:18:05.140955 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:18:05.141280 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.131279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:49415)
0808 02:18:05.131454 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:05.131564 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:05.132057 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:05.132294 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:05.132298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:05.133053 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:05.133323 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.133330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.134124 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.134124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.134918 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.134920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.135440 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.135441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.135671 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.136237 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:05.136248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:05.137921 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:05.139663 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:05.139671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:05.139674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:05.139903 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:05.140152 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:05.140153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:05.140154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:05.140270 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:05.140541 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:05.140549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:05.140697 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 02:18:05.140803 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:18:05.140982 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260808 02:18:05.141952 8924 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42207
I20260808 02:18:05.142083 8924 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:05.142304 8924 heartbeater.cc:511] Master 127.28.177.190:42207 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:18:05.142900 8629 ts_manager.cc:194] Registered new tserver with Master: c5477fe34a9b486d97613e494f8967c1 (127.28.177.129:44005)
I20260808 02:18:05.143563 8629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:49415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:18:05.163534 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42207
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:05.253006 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:05.253217 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:05.253240 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:05.254447 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:05.254496 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:05.254522 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:18:05.255991 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:05.256255 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:05.256505 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:05.262089 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.262095 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.262190 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:05.262244 8929 server_base.cc:1034] running on GCE node
I20260808 02:18:05.262562 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:05.262826 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:05.263990 8929 hybrid_clock.cc:648] HybridClock initialized: now 1786155485263965 us; error 40 us; skew 500 ppm
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:05.266649 8929 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:18:05.267087 8929 webserver.cc:460] Webserver started at http://127.28.177.130:32781/ using document root <none> and password file <none>
I20260808 02:18:05.267341 8929 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:05.267414 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:05.267571 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:05.268641 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "9fb83d0f2d0943769a9f6a9e07ac8bd6"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "a1554ae97c28c6c194c62a6e41e2f2b1"
server_key_iv: "5b899f215ade08ddbcd62cb651db43eb"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.269042 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb83d0f2d0943769a9f6a9e07ac8bd6"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "a1554ae97c28c6c194c62a6e41e2f2b1"
server_key_iv: "5b899f215ade08ddbcd62cb651db43eb"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.270516 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:05.271314 8952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.271489 8929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:05.271639 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "9fb83d0f2d0943769a9f6a9e07ac8bd6"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "a1554ae97c28c6c194c62a6e41e2f2b1"
server_key_iv: "5b899f215ade08ddbcd62cb651db43eb"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.271719 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:05.302409 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:05.304522 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:05.304710 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:05.304993 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:05.305343 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:05.305400 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.305444 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:05.305471 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.311659 8929 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:46853
I20260808 02:18:05.311738 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:46853 every 8 connection(s)
I20260808 02:18:05.312096 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:05.320351 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 8929
I20260808 02:18:05.320487 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:18:05.320725 29382 external_mini_cluster.cc:1428] Setting key 8b7f60c35602ecebbeec00446bc8d89b
I20260808 02:18:05.322768 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.312929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37795 (local address 127.28.177.190:42207)
0808 02:18:05.313007 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:05.313010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:05.313216 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:18:05.313760 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:05.313762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:05.313784 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:05.313852 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:05.314792 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.315281 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.316259 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.316451 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.316513 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.318885 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:05.318886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:05.318887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:05.318894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:05.320644 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:05.321473 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:05.321474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:05.321474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:05.321543 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:05.321952 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:05.321954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:05.321959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:05.322106 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:05.322192 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:05.322561 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 02:18:05.322668 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:18:05.323122 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.312913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:37795)
0808 02:18:05.313089 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:05.313132 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:05.313601 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:05.313938 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:05.313941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:05.314517 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:05.314712 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.314717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.315368 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.315369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.316170 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.316172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.316555 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.316557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.316678 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.317249 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:05.317258 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:05.318796 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:05.320755 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:05.320764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:05.320767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:05.321110 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:05.321659 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:05.321660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:05.321660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:05.321877 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:05.322197 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:05.322205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:05.322376 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 02:18:05.322516 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 02:18:05.322718 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260808 02:18:05.323721 9128 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42207
I20260808 02:18:05.323848 9128 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:05.324052 9128 heartbeater.cc:511] Master 127.28.177.190:42207 requested a full tablet report, sending...
I20260808 02:18:05.324522 8629 ts_manager.cc:194] Registered new tserver with Master: 9fb83d0f2d0943769a9f6a9e07ac8bd6 (127.28.177.130:46853)
I20260808 02:18:05.325019 8629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:37795
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:18:05.353261 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42207
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:05.442162 9136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:05.442411 9136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:05.442453 9136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:05.443626 9136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:05.443696 9136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:05.443742 9136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:18:05.445240 9136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:05.445526 9136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:05.445771 9136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:05.451066 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:05.451201 9136 server_base.cc:1034] running on GCE node
W20260808 02:18:05.451090 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.451130 9142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:05.451650 9136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:05.451933 9136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:05.453217 9136 hybrid_clock.cc:648] HybridClock initialized: now 1786155485453212 us; error 74 us; skew 500 ppm
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:05.455946 9136 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:18:05.456403 9136 webserver.cc:460] Webserver started at http://127.28.177.131:45767/ using document root <none> and password file <none>
I20260808 02:18:05.456676 9136 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:05.456725 9136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:05.456923 9136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:05.458091 9136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "514d81c1515449ad86f6d0c501d97798"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "53952c2797479e2545643fb8e3c55e19"
server_key_iv: "70fb77a79e3e3d3f632acb163f76ac02"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.458494 9136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "514d81c1515449ad86f6d0c501d97798"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "53952c2797479e2545643fb8e3c55e19"
server_key_iv: "70fb77a79e3e3d3f632acb163f76ac02"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.459959 9136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:18:05.460759 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.461001 9136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:05.461122 9136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "514d81c1515449ad86f6d0c501d97798"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "53952c2797479e2545643fb8e3c55e19"
server_key_iv: "70fb77a79e3e3d3f632acb163f76ac02"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.461215 9136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:05.483127 9136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:05.485386 9136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:05.485572 9136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:05.485836 9136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:05.486189 9136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:05.486246 9136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.486289 9136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:05.486318 9136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.492515 9136 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:42041
I20260808 02:18:05.492607 9323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:42041 every 8 connection(s)
I20260808 02:18:05.492980 9136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:05.499468 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 9136
I20260808 02:18:05.499580 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:18:05.499807 29382 external_mini_cluster.cc:1428] Setting key 79bf060dbd6db40f6f4e1592c9ef7433
I20260808 02:18:05.502822 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.493829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:35871 (local address 127.28.177.190:42207)
0808 02:18:05.493900 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:05.493902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:05.494106 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:18:05.494571 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:05.494572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:05.494584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:05.494648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:05.495561 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.496062 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.497120 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.497310 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.497371 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.499684 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:05.499686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:05.499686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:05.499694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:05.501413 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:05.501818 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:05.501819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:05.501819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:05.501871 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:05.502132 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:05.502133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:05.502134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:05.502268 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:05.502341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:05.502594 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 02:18:05.502673 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:18:05.502854 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.493798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:35871)
0808 02:18:05.493952 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:05.493995 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:05.494475 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:05.494700 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:05.494703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:05.495276 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:05.495472 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.495478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.496150 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.496151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.497016 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.497019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.497390 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.497391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.497493 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.497979 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:05.497987 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:05.499542 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:05.501489 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:05.501496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:05.501497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:05.501748 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:05.501932 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:05.501932 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:05.501933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:05.502035 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:05.502338 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:05.502342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:05.502458 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 02:18:05.502561 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:18:05.502687 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260808 02:18:05.503507 9324 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42207
I20260808 02:18:05.503603 9324 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:05.503749 9324 heartbeater.cc:511] Master 127.28.177.190:42207 requested a full tablet report, sending...
I20260808 02:18:05.504194 8629 ts_manager.cc:194] Registered new tserver with Master: 514d81c1515449ad86f6d0c501d97798 (127.28.177.131:42041)
I20260808 02:18:05.504526 8629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:35871
I20260808 02:18:05.511608 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab.
Entry for principal oryx/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab.
WARNING: no policy specified for oryx/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab.
Entry for principal oryx/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab.
WARNING: no policy specified for oryx/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab.
Entry for principal oryx/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab.
I20260808 02:18:05.592584 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 8722
I20260808 02:18:05.607522 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 8929
I20260808 02:18:05.619984 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 9136
I20260808 02:18:05.627476 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 8582
I20260808 02:18:05.640254 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42207
--webserver_interface=127.28.177.190
--webserver_port=46101
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:05.724444 9336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:05.724723 9336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:05.724763 9336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:05.726004 9336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:05.726085 9336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:05.726114 9336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:05.726136 9336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:05.726163 9336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:05.727764 9336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:05.728060 9336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:05.728319 9336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:05.733603 9345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.733614 9348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.733696 9342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:05.733927 9336 server_base.cc:1034] running on GCE node
I20260808 02:18:05.734320 9336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:05.734647 9336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:05.735826 9336 hybrid_clock.cc:648] HybridClock initialized: now 1786155485735801 us; error 47 us; skew 500 ppm
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:05.738896 9336 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:18:05.739358 9336 webserver.cc:460] Webserver started at http://127.28.177.190:46101/ using document root <none> and password file <none>
I20260808 02:18:05.739580 9336 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:05.739638 9336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:05.741175 9336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:05.742008 9357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.742220 9336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:05.742342 9336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "50678c902f61437caa599b402619958b"
format_stamp: "Formatted at 2026-08-08 02:18:04 on dist-test-slave-3h5h"
server_key: "6c57200ec1ec0024b2792b8d3c36960c"
server_key_iv: "30dbae696c45986b0a6fbada66302938"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.742745 9336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:05.751093 9336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:05.753546 9336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:05.753731 9336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:05.757825 9336 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42207
I20260808 02:18:05.757861 9439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42207 every 8 connection(s)
I20260808 02:18:05.758270 9336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:18:05.760025 9441 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:18:05.760728 9441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Bootstrap starting.
I20260808 02:18:05.766110 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 9336
I20260808 02:18:05.766492 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:44005
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:42207
--webserver_port=42295
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:18:05.770160 9441 log.cc:824] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:05.771984 9441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:18:05.772305 9441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Bootstrap complete.
I20260808 02:18:05.774730 9441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:05.774982 9441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50678c902f61437caa599b402619958b, State: Initialized, Role: FOLLOWER
I20260808 02:18:05.775192 9441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:05.775287 9441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:18:05.775342 9441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:18:05.775390 9441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:18:05.776443 9441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:05.776583 9441 leader_election.cc:302] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50678c902f61437caa599b402619958b; no voters:
I20260808 02:18:05.776805 9441 leader_election.cc:288] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:18:05.776861 9446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:18:05.777000 9446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [term 2 LEADER]: Becoming Leader. State: Replica: 50678c902f61437caa599b402619958b, State: Running, Role: LEADER
I20260808 02:18:05.777110 9441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:18:05.777092 9446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } }
I20260808 02:18:05.777376 9446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50678c902f61437caa599b402619958b. Latest consensus state: current_term: 2 leader_uuid: "50678c902f61437caa599b402619958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } } }
I20260808 02:18:05.777379 9447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "50678c902f61437caa599b402619958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50678c902f61437caa599b402619958b" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42207 } } }
I20260808 02:18:05.777503 9447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:05.777494 9446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:05.777802 9453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:18:05.778337 9453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:18:05.778465 9453 catalog_manager.cc:1259] Loaded cluster ID: 94714c5ec5404ab6a55e05e4a7319f55
I20260808 02:18:05.778658 9453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:18:05.779886 9453 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:18:05.780074 9453 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 50678c902f61437caa599b402619958b: Loaded TSK: 0
I20260808 02:18:05.780298 9453 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:18:05.853907 9444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:05.854149 9444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:05.854176 9444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:05.855484 9444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:05.855532 9444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:05.855562 9444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:18:05.857206 9444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.129.keytab
--principal=oryx/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:44005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=42295
--tserver_master_addrs=127.28.177.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:05.857465 9444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:05.857740 9444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:05.863221 9477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.863246 9478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:05.863548 9480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:05.863653 9444 server_base.cc:1034] running on GCE node
I20260808 02:18:05.863869 9444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:05.864123 9444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:05.865270 9444 hybrid_clock.cc:648] HybridClock initialized: now 1786155485865243 us; error 44 us; skew 500 ppm
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155485, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:05.868247 9444 init.cc:377] Logged in from keytab as oryx/127.28.177.129@KRBTEST.COM (short username oryx)
I20260808 02:18:05.868853 9444 webserver.cc:460] Webserver started at http://127.28.177.129:42295/ using document root <none> and password file <none>
I20260808 02:18:05.869150 9444 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:05.869246 9444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:05.871234 9444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:18:05.871975 9488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.872202 9444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:05.872309 9444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "c5477fe34a9b486d97613e494f8967c1"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "9bd8877661841355ba8c9842cc0fdaad"
server_key_iv: "738b823d11287a79bc362de123bbc17f"
server_key_version: "encryptionkey@0"
I20260808 02:18:05.872788 9444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:05.886648 9444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:05.889262 9444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:05.889436 9444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:05.889725 9444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:05.890125 9444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:05.890175 9444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.890208 9444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:05.890232 9444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:05.897794 9444 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:44005
I20260808 02:18:05.897833 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:44005 every 8 connection(s)
I20260808 02:18:05.898301 9444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:18:05.903177 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 9444
I20260808 02:18:05.903498 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:46853
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:42207
--webserver_port=32781
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:05.907955 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.899141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:53813)
0808 02:18:05.899336 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:05.899387 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:05.899886 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:05.900347 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:05.900351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:05.901028 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:05.901240 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.901265 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.902424 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.902425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.903462 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.903465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.904320 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.904321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.904472 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.905080 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:05.905091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:05.907717 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
W20260808 02:18:05.908154 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:05.899192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53813 (local address 127.28.177.190:42207)
0808 02:18:05.899425 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:05.899435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:05.899471 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:18:05.900108 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:05.900113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:05.900165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:05.900308 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:05.901396 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.902308 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.903580 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.904208 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.904324 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.907908 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:53813: BlockingRecv error: recv got EOF from 127.28.177.129:53813 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":116,"threads_started":1}
W20260808 02:18:05.908929 9650 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:42207 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.177.190:42207: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:05.915063 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.909226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:56525)
0808 02:18:05.909292 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:05.909302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:05.909339 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:05.909503 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:05.909504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:05.909662 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:05.909730 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.909733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.910727 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.910729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.911581 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.911584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.912145 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.912146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.912213 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.912739 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:05.912744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:05.914990 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:18:05.915556 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:05.909279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:56525 (local address 127.28.177.190:42207)
0808 02:18:05.909323 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:05.909325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:05.909335 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:18:05.909412 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:05.909414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:05.909428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:05.909489 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:05.909821 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.910631 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.911679 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.912046 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.912127 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.915473 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:56525: BlockingRecv error: recv got EOF from 127.28.177.129:56525 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:05 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:05.920815 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:05.915488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:56417)
0808 02:18:05.915710 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:05.915720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:05.915756 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:05.915936 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:05.915938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:05.916076 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:05.916115 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.916117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.917077 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.917078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.917809 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:05.917812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.918284 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.918285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:05.918335 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.918814 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:05.918819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:05.920747 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207}
W20260808 02:18:05.920990 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:05.915574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:56417 (local address 127.28.177.190:42207)
0808 02:18:05.915806 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:05.915809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:05.915818 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:05.915841 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:05.915842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:05.915853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:05.915922 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:05.916196 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.916992 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.917905 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:05.918205 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:05.918271 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:05.920919 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:56417: BlockingRecv error: recv got EOF from 127.28.177.129:56417 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
W20260808 02:18:05.921116 9650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:18:05.993690 9655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:05.993968 9655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:05.994010 9655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:05.995226 9655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:05.995293 9655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:05.995340 9655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:18:05.996910 9655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.130.keytab
--principal=oryx/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:46853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=32781
--tserver_master_addrs=127.28.177.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:05.997182 9655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:05.997426 9655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:06.002928 9662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:06.002928 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:06.002969 9661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:06.003268 9655 server_base.cc:1034] running on GCE node
I20260808 02:18:06.003484 9655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:06.003710 9655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:06.004926 9655 hybrid_clock.cc:648] HybridClock initialized: now 1786155486004895 us; error 40 us; skew 500 ppm
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155486, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:06.007526 9655 init.cc:377] Logged in from keytab as oryx/127.28.177.130@KRBTEST.COM (short username oryx)
I20260808 02:18:06.007997 9655 webserver.cc:460] Webserver started at http://127.28.177.130:32781/ using document root <none> and password file <none>
I20260808 02:18:06.008240 9655 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:06.008301 9655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:06.009833 9655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:06.010661 9678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:06.010860 9655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:06.010977 9655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "9fb83d0f2d0943769a9f6a9e07ac8bd6"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "a1554ae97c28c6c194c62a6e41e2f2b1"
server_key_iv: "5b899f215ade08ddbcd62cb651db43eb"
server_key_version: "encryptionkey@0"
I20260808 02:18:06.011477 9655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:06.030256 9655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:06.032663 9655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:06.032826 9655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:06.033162 9655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:06.033488 9655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:06.033526 9655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:06.033550 9655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:06.033601 9655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:06.039846 9655 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:46853
I20260808 02:18:06.039928 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:46853 every 8 connection(s)
I20260808 02:18:06.040313 9655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:18:06.041061 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 9655
I20260808 02:18:06.041450 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:42041
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:42207
--webserver_port=45767
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.048648 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.041154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:47421)
0808 02:18:06.041312 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.041355 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.041840 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.042121 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.042125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.042722 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.042922 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.042926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.043745 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.043747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.044560 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.044562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.045067 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.045068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.045205 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.045857 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.045866 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.048491 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260808 02:18:06.048728 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.041179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:47421 (local address 127.28.177.190:42207)
0808 02:18:06.041263 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.041266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.041456 (+ 190us) server_negotiation.cc:407] Connection header received
0808 02:18:06.041944 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.041946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.041964 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.042049 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.043037 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.043643 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.044668 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.044964 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.045052 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.048644 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:47421: BlockingRecv error: recv got EOF from 127.28.177.130:47421 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 02:18:06.048918 9842 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:42207 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.177.190:42207: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.053687 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.049144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:37661)
0808 02:18:06.049199 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.049209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.049251 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.049390 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.049391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.049530 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.049573 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.049575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.050254 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.050255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.050778 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.050780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.051157 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.051157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.051231 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.051726 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.051730 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.053621 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 02:18:06.053808 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.049212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37661 (local address 127.28.177.190:42207)
0808 02:18:06.049263 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.049265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.049279 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:18:06.049305 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.049305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.049315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.049388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.049703 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.050173 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.050866 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.051064 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.051155 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.053748 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:37661: BlockingRecv error: recv got EOF from 127.28.177.130:37661 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.058248 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.053966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:33337)
0808 02:18:06.054055 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.054065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.054133 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.054279 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.054280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.054403 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.054430 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.054431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.055051 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.055053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.055602 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.055604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.055979 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.055980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.056032 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.056470 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.056474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.058162 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260808 02:18:06.058297 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.054072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:33337 (local address 127.28.177.190:42207)
0808 02:18:06.054135 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.054137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.054146 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:06.054191 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.054192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.054201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.054271 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.054514 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.054972 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.055721 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.055906 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.055983 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.058236 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:33337: BlockingRecv error: recv got EOF from 127.28.177.130:33337 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 02:18:06.058388 9842 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:18:06.131366 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:06.131597 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:06.131624 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:06.133035 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:06.133085 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:06.133116 9848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:18:06.134805 9848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/krb5kdc/oryx_127.28.177.131.keytab
--principal=oryx/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:42041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=45767
--tserver_master_addrs=127.28.177.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:06.135205 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:06.135514 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:06.141381 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:06.141387 9857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:06.141425 9848 server_base.cc:1034] running on GCE node
W20260808 02:18:06.141398 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:06.141927 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:06.142163 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:06.143325 9848 hybrid_clock.cc:648] HybridClock initialized: now 1786155486143309 us; error 32 us; skew 500 ppm
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155486, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:06.146198 9848 init.cc:377] Logged in from keytab as oryx/127.28.177.131@KRBTEST.COM (short username oryx)
I20260808 02:18:06.146781 9848 webserver.cc:460] Webserver started at http://127.28.177.131:45767/ using document root <none> and password file <none>
I20260808 02:18:06.147058 9848 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:06.147150 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:06.148813 9848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:18:06.149605 9867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:06.149798 9848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:18:06.149916 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "514d81c1515449ad86f6d0c501d97798"
format_stamp: "Formatted at 2026-08-08 02:18:05 on dist-test-slave-3h5h"
server_key: "53952c2797479e2545643fb8e3c55e19"
server_key_iv: "70fb77a79e3e3d3f632acb163f76ac02"
server_key_version: "encryptionkey@0"
I20260808 02:18:06.150292 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:06.159524 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:06.161661 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:06.161852 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:06.162109 9848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:06.162451 9848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:06.162511 9848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:06.162557 9848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:06.162609 9848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:06.169401 9848 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:42041
I20260808 02:18:06.169486 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:42041 every 8 connection(s)
I20260808 02:18:06.169888 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.178013 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 9848
W20260808 02:18:06.178335 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.170700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:44145 (local address 127.28.177.190:42207)
0808 02:18:06.170755 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.170757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.171045 (+ 288us) server_negotiation.cc:407] Connection header received
0808 02:18:06.171607 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.171609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.171624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.171687 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.172733 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.173299 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.174383 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.174586 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.174702 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.178271 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:44145: BlockingRecv error: recv got EOF from 127.28.177.131:44145 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:18:06.178525 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.170693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:44145)
0808 02:18:06.170904 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.170944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.171490 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.171757 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.171761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.172371 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.172622 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.172628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.173407 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.173409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.174259 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.174261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.174714 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.174715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.174855 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.175344 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.175352 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.178198 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
W20260808 02:18:06.178992 10024 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:42207 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.177.190:42207: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.184017 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.179239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:56355)
0808 02:18:06.179295 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.179307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.179353 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.179523 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.179523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.179645 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.179696 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.179697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.180424 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.180425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.181029 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.181031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.181425 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.181426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.181494 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.182006 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.182011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.183934 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 02:18:06.184191 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.179298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:56355 (local address 127.28.177.190:42207)
0808 02:18:06.179356 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.179358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.179397 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:18:06.179420 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.179421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.179435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.179523 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.179777 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.180329 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.181128 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.181338 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.181426 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.184109 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:56355: BlockingRecv error: recv got EOF from 127.28.177.131:56355 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:18:06.187110 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.178711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.28.177.190:42207)
0808 02:18:06.178919 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.178922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.178931 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:06.178956 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.178957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.178968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.179039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.179887 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.180342 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.181296 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.181494 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.181589 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.182964 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:06.182970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:06.182970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:06.182985 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:06.185083 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:06.185501 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:06.185504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:06.185504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:06.185602 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:06.186420 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:06.186421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:06.186421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:06.186668 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:06.186774 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:06.186938 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 02:18:06.187026 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.189766 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.185197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:58017)
0808 02:18:06.185254 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.185264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.185302 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.185419 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.185419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.185562 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.185594 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.185596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.186242 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.186243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.186911 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.186914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.187424 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.187425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.187476 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.187885 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.187889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.189684 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 02:18:06.189821 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.185244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:58017 (local address 127.28.177.190:42207)
0808 02:18:06.185284 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.185285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.185300 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:18:06.185363 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.185363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.185372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.185412 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.185677 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.186156 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.187008 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.187298 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.187344 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.189758 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:58017: BlockingRecv error: recv got EOF from 127.28.177.131:58017 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 02:18:06.190053 10024 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:06 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:06.926503 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:06.921572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:48383)
0808 02:18:06.921735 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:06.921746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:06.921807 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:06.921940 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:06.921941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:06.922111 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:06.922183 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.922185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.922934 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.922935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.923494 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:06.923496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.923848 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.923848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:06.923892 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.924321 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:06.924325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:06.926435 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
W20260808 02:18:06.926717 10030 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:06.921665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:48383 (local address 127.28.177.190:42207)
0808 02:18:06.921806 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:06.921808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:06.921818 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:18:06.921849 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:06.921850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:06.921861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:06.921924 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:06.922262 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.922832 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.923579 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:06.923776 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:06.923834 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:06.926620 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:48383: BlockingRecv error: recv got EOF from 127.28.177.129:48383 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
Aug 08 02:18:07 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:07 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:07.063519 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:07.058806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:33581)
0808 02:18:07.058947 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:07.058965 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:07.059025 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:07.059213 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:07.059214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:07.059391 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:07.059446 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:07.059447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.060112 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.060113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:07.060666 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:07.060668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.061101 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.061102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:07.061144 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:07.061573 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:07.061578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:07.063447 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260808 02:18:07.063588 10030 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:07.058909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:33581 (local address 127.28.177.190:42207)
0808 02:18:07.058981 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:07.058982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:07.059009 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:18:07.059086 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:07.059087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:07.059099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:07.059187 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:07.059510 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.059994 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.060740 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.060995 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.061052 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:07.063542 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:33581: BlockingRecv error: recv got EOF from 127.28.177.130:33581 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 08 02:18:07 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:07 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:07.195878 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:07.190670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:51795)
0808 02:18:07.190833 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:07.190844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:07.190938 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:07.191112 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:07.191113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:07.191305 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:07.191351 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:07.191353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.192034 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.192035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:07.192593 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:07.192595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.193052 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.193054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:07.193132 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:07.193651 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:07.193656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:07.195752 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
W20260808 02:18:07.195937 10030 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:07.190713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:51795 (local address 127.28.177.190:42207)
0808 02:18:07.190786 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:07.190789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:07.190919 (+ 130us) server_negotiation.cc:407] Connection header received
0808 02:18:07.191005 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:07.191006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:07.191023 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:07.191087 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:07.191440 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.191935 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.192696 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.192962 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.193049 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:07.195845 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:51795: BlockingRecv error: recv got EOF from 127.28.177.131:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 02:18:07 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:07 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:07.932174 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:07.927009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:33937)
0808 02:18:07.927216 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:07.927236 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:07.927295 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:07.927479 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:07.927480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:07.927671 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:07.927723 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:07.927725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.928514 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.928515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:07.929093 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:07.929096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.929484 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.929485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:07.929527 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:07.930175 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:07.930180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:07.932077 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
W20260808 02:18:07.932430 10035 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:07.927100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:33937 (local address 127.28.177.190:42207)
0808 02:18:07.927262 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:07.927264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:07.927277 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:18:07.927359 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:07.927359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:07.927373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:07.927444 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:07.927869 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.928430 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.929183 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:07.929388 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:07.929460 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:07.932350 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:33937: BlockingRecv error: recv got EOF from 127.28.177.129:33937 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
Aug 08 02:18:08 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:08 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:08.068992 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:08.064122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:54977)
0808 02:18:08.064288 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:08.064299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:08.064361 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:08.064509 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:08.064510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:08.064673 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:08.064723 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:08.064724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.065482 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.065484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:08.066081 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:08.066085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.066421 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.066422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:08.066461 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:08.066992 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:08.066998 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:08.068879 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
W20260808 02:18:08.069226 10035 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:08.064230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:54977 (local address 127.28.177.190:42207)
0808 02:18:08.064288 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:08.064289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:08.064346 (+ 57us) server_negotiation.cc:407] Connection header received
0808 02:18:08.064424 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:08.064425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:08.064437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:08.064482 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:08.064793 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.065415 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.066167 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.066355 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.066409 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:08.069152 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:54977: BlockingRecv error: recv got EOF from 127.28.177.130:54977 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 08 02:18:08 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:08 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:08.201998 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:08.196495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:34109)
0808 02:18:08.196668 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:08.196689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:08.196765 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:08.197027 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:08.197028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:08.197245 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:08.197295 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:08.197296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.198449 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.198451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:08.199012 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:08.199014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.199421 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.199422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:08.199466 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:08.199955 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:08.199960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:08.201937 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
W20260808 02:18:08.202164 10035 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:08.196755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:34109 (local address 127.28.177.190:42207)
0808 02:18:08.196870 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:08.196873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:08.196884 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:18:08.196917 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:08.196919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:08.196939 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:08.197005 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:08.197401 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.198306 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.199121 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.199341 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.199413 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:08.202056 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:34109: BlockingRecv error: recv got EOF from 127.28.177.131:34109 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 02:18:08 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:08 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:08.938606 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:08.932733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:38007)
0808 02:18:08.932988 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:08.933012 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:08.933240 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:08.933432 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:08.933433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:08.933617 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:08.933672 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:08.933684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.934643 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.934645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:08.935562 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:08.935565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.935916 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.935916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:08.935958 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:08.936599 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:08.936609 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:08.938513 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":172,"threads_started":1}
W20260808 02:18:08.938681 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:08.932826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:38007 (local address 127.28.177.190:42207)
0808 02:18:08.932994 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:08.932997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:08.933120 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:18:08.933323 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:08.933324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:08.933342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:08.933420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:08.933767 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.934522 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.935655 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:08.935853 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:08.935907 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:08.938627 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:38007: BlockingRecv error: recv got EOF from 127.28.177.129:38007 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
Aug 08 02:18:09 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:09 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:18:09.074563 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:09.069567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:49075 (local address 127.28.177.190:42207)
0808 02:18:09.069625 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:09.069627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:09.069733 (+ 106us) server_negotiation.cc:407] Connection header received
0808 02:18:09.069807 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:09.069807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:09.069820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:09.069873 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:09.070178 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.070667 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.071561 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.071760 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.071829 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:09.074508 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:49075: BlockingRecv error: recv got EOF from 127.28.177.130:49075 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 02:18:09.074577 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:09.069492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:49075)
0808 02:18:09.069653 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:09.069666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:09.069738 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:09.069889 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:09.069890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:09.070062 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:09.070119 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:09.070120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.070786 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.070788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:09.071453 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:09.071457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.071863 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.071864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:09.071918 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:09.072391 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:09.072395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:09.074417 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":120,"threads_started":1}
Aug 08 02:18:09 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:09 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:09.207705 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:09.202591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:57977)
0808 02:18:09.202744 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:09.202757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:09.202821 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:09.202988 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:09.202989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:09.203166 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:09.203217 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:09.203218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.203897 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.203898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:09.204493 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:09.204495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.204911 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.204913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:09.204979 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:09.205442 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:09.205447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:09.207631 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20260808 02:18:09.207875 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:09.202782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:57977 (local address 127.28.177.190:42207)
0808 02:18:09.202840 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:09.202842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:09.202884 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:18:09.202907 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:09.202908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:09.202919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:09.202966 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:09.203287 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.203802 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.204574 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.204769 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.204822 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:09.207741 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:57977: BlockingRecv error: recv got EOF from 127.28.177.131:57977 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 02:18:09 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:09 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:09.949795 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:09.939185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:42251)
0808 02:18:09.939355 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:09.939366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:09.939426 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:09.939558 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:09.939559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:09.939734 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:09.939785 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:09.939786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.940544 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.940545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:09.941117 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:09.941119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.941575 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.941576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:09.941618 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:09.947548 (+ 5930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:09.947554 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:09.949669 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
W20260808 02:18:09.949922 10046 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:09.939295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:42251 (local address 127.28.177.190:42207)
0808 02:18:09.939432 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:09.939434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:09.939448 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:18:09.939472 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:09.939473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:09.939483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:09.939546 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:09.939888 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.940460 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.941226 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:09.941491 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:09.941559 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:09.949792 (+ 8233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:42251: BlockingRecv error: recv got EOF from 127.28.177.129:42251 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
Aug 08 02:18:10 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:10 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:10.081279 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:10.075184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:59833)
0808 02:18:10.075390 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:10.075404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:10.075513 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:10.075709 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:10.075710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:10.076079 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:10.076154 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:10.076158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.077060 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.077061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:10.078033 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:10.078037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.078510 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.078512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:10.078573 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:10.079107 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:10.079112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:10.081170 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
W20260808 02:18:10.081347 10046 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:10.075310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:59833 (local address 127.28.177.190:42207)
0808 02:18:10.075391 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:10.075393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:10.075510 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:18:10.075606 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:10.075607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:10.075629 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:10.075702 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:10.076243 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.076910 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.078155 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.078383 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.078525 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:10.081276 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:59833: BlockingRecv error: recv got EOF from 127.28.177.130:59833 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 02:18:10 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:10 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:10.213657 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:10.208306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:35653)
0808 02:18:10.208528 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:10.208542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:10.208614 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:10.208818 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:10.208819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:10.209085 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:10.209194 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:10.209196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.209906 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.209907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:10.210448 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:10.210450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.210811 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.210812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:10.210872 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:10.211395 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:10.211400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:10.213563 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":143,"threads_started":1}
W20260808 02:18:10.213760 10046 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:10.208463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:35653 (local address 127.28.177.190:42207)
0808 02:18:10.208532 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:10.208534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:10.208595 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:18:10.208689 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:10.208695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:10.208728 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:10.208788 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:10.209283 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.209824 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.210538 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.210736 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.210811 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:10.213694 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:35653: BlockingRecv error: recv got EOF from 127.28.177.131:35653 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 08 02:18:10 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:10 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:10.961223 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:10.950468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:54121)
0808 02:18:10.950646 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:10.950664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:10.950719 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:10.950873 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:10.950874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:10.951059 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:10.951148 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:10.951150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.951919 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.951921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:10.952769 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:10.952774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.953289 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.953290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:10.953337 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:10.959132 (+ 5795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:10.959144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:10.961152 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":131,"threads_started":1}
W20260808 02:18:10.961436 10050 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:10.950570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:54121 (local address 127.28.177.190:42207)
0808 02:18:10.950715 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:10.950716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:10.950728 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:18:10.950773 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:10.950774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:10.950785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:10.950857 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:10.951226 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.951819 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.952904 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:10.953204 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:10.953280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:10.961354 (+ 8074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:54121: BlockingRecv error: recv got EOF from 127.28.177.129:54121 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
Aug 08 02:18:11 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:11 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:11.087196 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:11.081943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:60425)
0808 02:18:11.082129 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:11.082141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:11.082202 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:11.082364 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:11.082364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:11.082556 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:11.082604 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:11.082607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.083410 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.083411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:11.083966 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:11.083967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.084450 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.084450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:11.084495 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:11.085020 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:11.085025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:11.087136 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
W20260808 02:18:11.087356 10050 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:11.082098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:60425 (local address 127.28.177.190:42207)
0808 02:18:11.082173 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:11.082175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:11.082189 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:18:11.082274 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:11.082275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:11.082289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:11.082349 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:11.082703 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.083296 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.084035 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.084382 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.084448 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:11.087270 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:60425: BlockingRecv error: recv got EOF from 127.28.177.130:60425 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 02:18:11 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:11 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:11.219348 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:11.214242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:59815)
0808 02:18:11.214405 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:11.214418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:11.214482 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:11.214700 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:11.214701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:11.214874 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:11.214922 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:11.214924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.215700 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.215701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:11.216535 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:11.216537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.216883 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.216884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:11.216940 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:11.217387 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:11.217392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:11.219288 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260808 02:18:11.219542 10050 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:11.214448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:59815 (local address 127.28.177.190:42207)
0808 02:18:11.214520 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:11.214522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:11.214531 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:11.214582 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:11.214583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:11.214612 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:11.214669 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:11.214996 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.215589 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.216599 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.216786 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.217005 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:11.219395 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:59815: BlockingRecv error: recv got EOF from 127.28.177.131:59815 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 02:18:11 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:11 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:11.967023 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:11.961783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:53815)
0808 02:18:11.961962 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:11.961976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:11.962058 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:11.962205 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:11.962207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:11.962377 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:11.962447 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:11.962449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.963150 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.963151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:11.963843 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:11.963846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.964227 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.964228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:11.964302 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:11.964783 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:11.964788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:11.966927 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
W20260808 02:18:11.967123 10056 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:11.961901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53815 (local address 127.28.177.190:42207)
0808 02:18:11.962056 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:11.962058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:11.962072 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:18:11.962107 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:11.962108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:11.962122 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:11.962186 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:11.962517 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.963055 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.963932 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:11.964141 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:11.964215 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:11.967024 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:53815: BlockingRecv error: recv got EOF from 127.28.177.129:53815 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
Aug 08 02:18:12 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:12 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:12.092705 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:12.087715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:60299)
0808 02:18:12.087881 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:12.087894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:12.087990 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:12.088154 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:12.088155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:12.088355 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:12.088445 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:12.088448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.089213 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.089214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:12.089752 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:12.089754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.090185 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.090187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:12.090246 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:12.090697 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:12.090702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:12.092643 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
W20260808 02:18:12.092906 10056 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:12.087871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:60299 (local address 127.28.177.190:42207)
0808 02:18:12.087945 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:12.087947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:12.087966 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:18:12.088060 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:12.088060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:12.088072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:12.088123 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:12.088545 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.089115 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.089856 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.090099 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.090189 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:12.092764 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:60299: BlockingRecv error: recv got EOF from 127.28.177.130:60299 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 02:18:12 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:12 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:12.225452 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:12.219935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:57939)
0808 02:18:12.220193 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:12.220209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:12.220300 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:12.220465 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:12.220466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:12.220678 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:12.220732 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:12.220734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.221515 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.221516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:12.222096 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:12.222098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.222468 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.222489 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:12.222534 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:12.223025 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:12.223031 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:12.225388 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":159,"threads_started":1}
W20260808 02:18:12.225566 10056 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:12.220177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:57939 (local address 127.28.177.190:42207)
0808 02:18:12.220256 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:12.220258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:12.220284 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:18:12.220364 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:12.220365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:12.220380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:12.220440 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:12.220812 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.221414 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.222182 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.222393 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.222459 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:12.225494 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:57939: BlockingRecv error: recv got EOF from 127.28.177.131:57939 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 02:18:12 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:12 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:12.972764 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:12.967670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:40203)
0808 02:18:12.967841 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:12.967853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:12.967908 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:12.968172 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:12.968174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:12.968361 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:12.968414 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:12.968415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.969148 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.969149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:12.969764 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:12.969767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.970176 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.970178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:12.970225 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:12.970713 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:12.970719 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:12.972701 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
W20260808 02:18:12.973032 10064 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:12.967839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:40203 (local address 127.28.177.190:42207)
0808 02:18:12.968010 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:12.968012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:12.968020 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:18:12.968044 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:12.968045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:12.968057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:12.968126 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:12.968497 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.969014 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.969852 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:12.970054 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:12.970110 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:12.972812 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:40203: BlockingRecv error: recv got EOF from 127.28.177.129:40203 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
Aug 08 02:18:13 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:13 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:13.098469 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:13.093238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:45225)
0808 02:18:13.093446 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:13.093459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:13.093583 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:13.093788 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:13.093789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:13.093988 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:13.094038 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:13.094040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.094732 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.094733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:13.095313 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:13.095316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.095706 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.095707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:13.095763 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:13.096280 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:13.096285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:13.098404 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260808 02:18:13.098531 10064 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:13.093346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:45225 (local address 127.28.177.190:42207)
0808 02:18:13.093430 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:13.093433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:13.093530 (+ 97us) server_negotiation.cc:407] Connection header received
0808 02:18:13.093669 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:13.093670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:13.093695 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:13.093767 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:13.094122 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.094639 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.095409 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.095620 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.095704 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:13.098483 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:45225: BlockingRecv error: recv got EOF from 127.28.177.130:45225 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 02:18:13 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:13 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:13.230796 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:13.226010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:59313)
0808 02:18:13.226159 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:13.226170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:13.226225 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:13.226379 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:13.226380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:13.226548 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:13.226604 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:13.226606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.227325 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.227327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:13.227857 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:13.227859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.228218 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.228219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:13.228292 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:13.228875 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:13.228882 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:13.230697 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
W20260808 02:18:13.230839 10064 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:13.226098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:59313 (local address 127.28.177.190:42207)
0808 02:18:13.226187 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:13.226190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:13.226208 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:18:13.226286 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:13.226287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:13.226301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:13.226360 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:13.226672 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.227245 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.227928 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.228108 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.228217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:13.230775 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:59313: BlockingRecv error: recv got EOF from 127.28.177.131:59313 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 02:18:13 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:13 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:13.979079 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:13.973624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:45271)
0808 02:18:13.973795 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:13.973808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:13.973871 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:13.974037 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:13.974038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:13.974220 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:13.974271 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:13.974273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.975336 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.975338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:13.975926 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:13.975928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.976261 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.976262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:13.976304 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:13.976881 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:13.976887 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:13.979018 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
W20260808 02:18:13.979255 10068 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:13.973768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:45271 (local address 127.28.177.190:42207)
0808 02:18:13.973902 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:13.973905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:13.973913 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:18:13.973938 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:13.973939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:13.973952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:13.974020 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:13.974345 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.975241 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.975998 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:13.976193 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:13.976251 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:13.979139 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:45271: BlockingRecv error: recv got EOF from 127.28.177.129:45271 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
Aug 08 02:18:14 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:14 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:14.104235 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:14.099124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:46793)
0808 02:18:14.099278 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:14.099290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:14.099349 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:14.099634 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:14.099635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:14.099809 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:14.099860 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:14.099861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.100778 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.100779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:14.101337 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:14.101339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.101763 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.101764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:14.101817 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:14.102361 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:14.102366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:14.104173 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260808 02:18:14.104390 10068 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:14.099194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:46793 (local address 127.28.177.190:42207)
0808 02:18:14.099316 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:14.099318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:14.099332 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:18:14.099473 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:14.099474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:14.099489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:14.099557 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:14.099971 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.100695 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.101452 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.101671 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.101741 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:14.104306 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:46793: BlockingRecv error: recv got EOF from 127.28.177.130:46793 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 08 02:18:14 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:14 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:14.236394 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:14.231378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:42871)
0808 02:18:14.231541 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:14.231557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:14.231616 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:14.231833 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:14.231834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:14.232003 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:14.232053 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:14.232054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.232745 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.232745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:14.233313 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:14.233315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.233698 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.233699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:14.233753 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:14.234308 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:14.234315 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:14.236323 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
W20260808 02:18:14.236596 10068 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:14.231446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:42871 (local address 127.28.177.190:42207)
0808 02:18:14.231531 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:14.231533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:14.231642 (+ 109us) server_negotiation.cc:407] Connection header received
0808 02:18:14.231674 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:14.231675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:14.231695 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:14.231800 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:14.232154 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.232667 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.233393 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.233588 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.233649 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:14.236533 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:42871: BlockingRecv error: recv got EOF from 127.28.177.131:42871 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:18:14 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:14 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:14.989274 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:14.979631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:60145)
0808 02:18:14.979813 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:14.979828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:14.979908 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:14.980118 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:14.980119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:14.980299 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:14.980350 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:14.980352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.981419 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.981420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:14.982001 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:14.982003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.982383 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.982384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:14.982426 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:14.987085 (+ 4659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:14.987091 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:14.989209 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
W20260808 02:18:14.989518 10074 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:14.979775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:60145 (local address 127.28.177.190:42207)
0808 02:18:14.979927 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:14.979929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:14.979938 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:14.979964 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:14.979965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:14.979976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:14.980039 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:14.980488 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.981302 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.982108 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:14.982305 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:14.982370 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:14.989328 (+ 6958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:60145: BlockingRecv error: recv got EOF from 127.28.177.129:60145 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
Aug 08 02:18:15 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:15 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:15.110627 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:15.104778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:45511)
0808 02:18:15.105079 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:15.105091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:15.105164 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:15.105400 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:15.105402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:15.105608 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:15.105668 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:15.105670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.106596 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.106597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:15.107238 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:15.107240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.107614 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.107615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:15.107662 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:15.108152 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:15.108157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:15.110535 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
W20260808 02:18:15.110746 10074 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:15.104907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:45511 (local address 127.28.177.190:42207)
0808 02:18:15.104989 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:15.104991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:15.105217 (+ 226us) server_negotiation.cc:407] Connection header received
0808 02:18:15.105251 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:15.105252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:15.105267 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:15.105346 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:15.105789 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.106484 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.107336 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.107531 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.107592 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:15.110673 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:45511: BlockingRecv error: recv got EOF from 127.28.177.130:45511 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 02:18:15 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:15 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:15.242200 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:15.236960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:34611)
0808 02:18:15.237083 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:15.237095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:15.237212 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:15.237396 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:15.237397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:15.237570 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:15.237618 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:15.237619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.238363 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.238364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:15.238937 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:15.238939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.239304 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.239304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:15.239345 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:15.239857 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:15.239861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:15.242022 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
W20260808 02:18:15.242198 10074 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:15.237023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:34611 (local address 127.28.177.190:42207)
0808 02:18:15.237087 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:15.237089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:15.237223 (+ 134us) server_negotiation.cc:407] Connection header received
0808 02:18:15.237279 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:15.237280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:15.237296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:15.237368 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:15.237739 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.238273 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.239045 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.239233 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.239293 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:15.242128 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:34611: BlockingRecv error: recv got EOF from 127.28.177.131:34611 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 08 02:18:15 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:15 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:15.995661 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:15.990078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:48363)
0808 02:18:15.990293 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:15.990306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:15.990370 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:15.990539 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:15.990540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:15.990724 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:15.990775 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:15.990777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.991577 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.991579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:15.992277 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:15.992280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.992818 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.992819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:15.992898 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:15.993394 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:15.993399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:15.995550 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":148,"threads_started":1}
W20260808 02:18:15.995715 10080 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:15.990214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:48363 (local address 127.28.177.190:42207)
0808 02:18:15.990375 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:15.990378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:15.990389 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:18:15.990420 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:15.990421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:15.990433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:15.990513 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:15.990866 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.991485 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.992377 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:15.992736 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:15.992815 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:15.995635 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:48363: BlockingRecv error: recv got EOF from 127.28.177.129:48363 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
Aug 08 02:18:16 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:16 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:16.117621 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:16.111405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:36411)
0808 02:18:16.111623 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:16.111636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:16.111709 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:16.111863 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:16.111864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:16.112065 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:16.112124 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:16.112126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.112953 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.112955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:16.113929 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:16.113932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.114448 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.114450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:16.114535 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:16.115257 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:16.115264 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:16.117556 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
W20260808 02:18:16.117791 10080 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:16.111493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:36411 (local address 127.28.177.190:42207)
0808 02:18:16.111601 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:16.111603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:16.111695 (+ 92us) server_negotiation.cc:407] Connection header received
0808 02:18:16.111763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:16.111764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:16.111780 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:16.111838 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:16.112217 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.112815 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.114056 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.114360 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.114427 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:16.117692 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:36411: BlockingRecv error: recv got EOF from 127.28.177.130:36411 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 08 02:18:16 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:16 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:16.248613 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:16.242808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:58739)
0808 02:18:16.242961 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:16.242972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:16.243036 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:16.243291 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:16.243293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:16.243529 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:16.243583 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:16.243585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.244290 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.244291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:16.244901 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:16.244905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.245306 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.245307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:16.245376 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:16.245917 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:16.245923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:16.248462 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260808 02:18:16.248615 10080 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:16.242985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:58739 (local address 127.28.177.190:42207)
0808 02:18:16.243157 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:16.243159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:16.243166 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:18:16.243191 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:16.243192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:16.243205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:16.243278 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:16.243682 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.244192 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.245002 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.245202 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.245293 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:16.248559 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:58739: BlockingRecv error: recv got EOF from 127.28.177.131:58739 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Aug 08 02:18:16 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:17 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:17.001931 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:16.996229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:57569)
0808 02:18:16.996393 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:16.996405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:16.996454 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:16.996663 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:16.996664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:16.996921 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:16.996985 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:16.996988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.997912 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.997913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:16.998572 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:16.998576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.998971 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.998972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:16.999024 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:16.999506 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:16.999511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:17.001845 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
W20260808 02:18:17.002071 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:16.996309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:57569 (local address 127.28.177.190:42207)
0808 02:18:16.996444 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:16.996446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:16.996455 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:16.996516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:16.996517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:16.996529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:16.996592 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:16.997182 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.997701 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.998670 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:16.998871 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:16.998927 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:17.001963 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:57569: BlockingRecv error: recv got EOF from 127.28.177.129:57569 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
Aug 08 02:18:17 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:17 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:17.122953 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:17.118154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:47641)
0808 02:18:17.118306 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:17.118318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:17.118384 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:17.118560 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:17.118561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:17.118762 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:17.118810 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:17.118812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.119445 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.119446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:17.119979 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:17.119980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.120402 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.120403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:17.120450 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:17.120949 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:17.120955 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:17.122878 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
W20260808 02:18:17.123202 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:17.118230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:47641 (local address 127.28.177.190:42207)
0808 02:18:17.118289 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:17.118292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:17.118387 (+ 95us) server_negotiation.cc:407] Connection header received
0808 02:18:17.118483 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:17.118484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:17.118497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:17.118557 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:17.118878 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.119375 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.120109 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.120315 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.120384 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:17.123010 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:47641: BlockingRecv error: recv got EOF from 127.28.177.130:47641 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 08 02:18:17 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:17 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:17.254884 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:17.249210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:47661)
0808 02:18:17.249364 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:17.249377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:17.249455 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:17.249673 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:17.249674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:17.249870 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:17.249919 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:17.249921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.250939 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.250941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:17.251778 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:17.251781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.252225 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.252226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:17.252268 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:17.252782 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:17.252787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:17.254814 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
W20260808 02:18:17.255170 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:17.249305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:47661 (local address 127.28.177.190:42207)
0808 02:18:17.249373 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:17.249375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:17.249455 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:18:17.249562 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:17.249563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:17.249579 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:17.249649 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:17.250009 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.250805 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.251885 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:17.252142 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:17.252216 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:17.255118 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:47661: BlockingRecv error: recv got EOF from 127.28.177.131:47661 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 08 02:18:18 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:18 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:18.007988 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:18.002617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:43733)
0808 02:18:18.002811 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:18.002823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:18.002889 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:18.003040 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:18.003041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:18.003225 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:18.003278 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:18.003280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.004040 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.004042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:18.004644 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:18.004647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.005119 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.005121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:18.005194 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:18.005816 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:18.005840 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:18.007916 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
W20260808 02:18:18.008111 10090 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:18.002745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:43733 (local address 127.28.177.190:42207)
0808 02:18:18.002884 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:18.002886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:18.002897 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:18:18.002944 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:18.002945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:18.002957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:18.003019 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:18.003351 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.003918 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.004726 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.005028 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.005084 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:18.008021 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:43733: BlockingRecv error: recv got EOF from 127.28.177.129:43733 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Aug 08 02:18:18 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:18 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:18.128748 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:18.123645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:34493)
0808 02:18:18.123811 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:18.123823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:18.123900 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:18.124118 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:18.124119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:18.124296 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:18.124346 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:18.124347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.125291 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.125292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:18.125824 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:18.125827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.126196 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.126198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:18.126252 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:18.126747 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:18.126752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:18.128643 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
W20260808 02:18:18.128798 10090 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:18.123755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:34493 (local address 127.28.177.190:42207)
0808 02:18:18.123808 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:18.123811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:18.123900 (+ 89us) server_negotiation.cc:407] Connection header received
0808 02:18:18.123957 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:18.123958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:18.123970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:18.124033 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:18.124471 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.125210 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.125931 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.126133 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.126192 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:18.128731 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:34493: BlockingRecv error: recv got EOF from 127.28.177.130:34493 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 02:18:18 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:18 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:18.261006 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:18.255381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:52027)
0808 02:18:18.255636 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:18.255648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:18.255718 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:18.255912 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:18.255914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:18.256151 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:18.256206 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:18.256208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.256942 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.256944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:18.257593 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:18.257596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.258060 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.258061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:18.258137 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:18.258832 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:18.258840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:18.260923 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":199,"threads_started":1}
W20260808 02:18:18.261016 10090 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:18.255477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:52027 (local address 127.28.177.190:42207)
0808 02:18:18.255551 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:18.255554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:18.255725 (+ 171us) server_negotiation.cc:407] Connection header received
0808 02:18:18.255762 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:18.255763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:18.255779 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:18.255844 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:18.256294 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.256787 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.257697 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:18.257962 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:18.258021 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:18.260965 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:52027: BlockingRecv error: recv got EOF from 127.28.177.131:52027 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 02:18:19 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:19 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:19.013705 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:19.008611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:44607)
0808 02:18:19.008767 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:19.008784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:19.008857 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:19.009026 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:19.009026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:19.009221 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:19.009271 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:19.009273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.009991 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.009992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:19.010558 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:19.010560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.010973 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.010975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:19.011025 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:19.011580 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:19.011585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:19.013621 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
W20260808 02:18:19.013837 10094 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:19.008698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:44607 (local address 127.28.177.190:42207)
0808 02:18:19.008880 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:19.008882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:19.008892 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:18:19.008926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:19.008926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:19.008939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:19.009006 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:19.009381 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.009893 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.010662 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.010877 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.010940 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:19.013746 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:44607: BlockingRecv error: recv got EOF from 127.28.177.129:44607 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
Aug 08 02:18:19 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:19 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:19.134454 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:19.129275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:41311)
0808 02:18:19.129743 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:19.129755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:19.129824 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:19.129988 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:19.129989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:19.130194 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:19.130241 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:19.130243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.130948 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.130950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:19.131484 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:19.131485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.131875 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.131876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:19.131936 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:19.132455 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:19.132460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:19.134356 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":219,"threads_started":1}
W20260808 02:18:19.134533 10094 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:19.129492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:41311 (local address 127.28.177.190:42207)
0808 02:18:19.129722 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:19.129725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:19.129805 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:18:19.129884 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:19.129885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:19.129899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:19.129979 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:19.130331 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.130847 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.131576 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.131784 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.131865 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:19.134457 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:41311: BlockingRecv error: recv got EOF from 127.28.177.130:41311 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
Aug 08 02:18:19 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:19 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:19.267491 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:19.261609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:44911)
0808 02:18:19.261858 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:19.261874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:19.262077 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:19.262287 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:19.262288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:19.262525 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:19.262578 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:19.262580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.263414 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.263416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:19.264045 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:19.264047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.264558 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.264559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:19.264632 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:19.265132 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:19.265138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:19.267361 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":127,"threads_started":1}
W20260808 02:18:19.267553 10094 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:19.261743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:44911 (local address 127.28.177.190:42207)
0808 02:18:19.261838 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:19.261841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:19.262046 (+ 205us) server_negotiation.cc:407] Connection header received
0808 02:18:19.262156 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:19.262157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:19.262179 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:19.262277 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:19.262710 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.263288 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.264163 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:19.264437 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:19.264553 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:19.267471 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:44911: BlockingRecv error: recv got EOF from 127.28.177.131:44911 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 02:18:20 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:20 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:20.024677 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:20.014342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:55515)
0808 02:18:20.014504 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:20.014517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:20.014583 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:20.014708 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:20.014708 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:20.014892 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:20.014941 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:20.014942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.015787 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.015788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:20.016422 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:20.016425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.016823 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.016824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:20.016930 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:20.022340 (+ 5410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:20.022346 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:20.024611 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20260808 02:18:20.024915 10098 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:20.014449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:55515 (local address 127.28.177.190:42207)
0808 02:18:20.014568 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:20.014570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:20.014581 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:18:20.014630 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:20.014630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:20.014641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:20.014702 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:20.015091 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.015675 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.016530 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.016747 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.016813 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:20.024801 (+ 7988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:55515: BlockingRecv error: recv got EOF from 127.28.177.129:55515 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
Aug 08 02:18:20 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:20 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:20.140548 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:20.135145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:53047)
0808 02:18:20.135313 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:20.135324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:20.135390 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:20.135549 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:20.135550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:20.135749 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:20.135801 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:20.135802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.136480 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.136482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:20.137251 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:20.137255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.137728 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.137730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:20.137782 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:20.138273 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:20.138279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:20.140430 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
W20260808 02:18:20.140606 10098 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:20.135263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:53047 (local address 127.28.177.190:42207)
0808 02:18:20.135318 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:20.135320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:20.135375 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:18:20.135443 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:20.135444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:20.135454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:20.135512 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:20.135885 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.136383 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.137368 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.137626 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.137694 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:20.140541 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:53047: BlockingRecv error: recv got EOF from 127.28.177.130:53047 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 08 02:18:20 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:20 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.131@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:20.273725 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:20.268140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.131:46631)
0808 02:18:20.268289 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:20.268302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:20.268397 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:20.268593 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:20.268595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:20.268787 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:20.268876 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:20.268878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.269706 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.269707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:20.270331 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:20.270334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.270748 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.270749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:20.270799 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:20.271411 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:20.271416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:20.273648 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260808 02:18:20.273815 10098 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:20.268236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:46631 (local address 127.28.177.190:42207)
0808 02:18:20.268335 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:20.268337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:20.268393 (+ 56us) server_negotiation.cc:407] Connection header received
0808 02:18:20.268461 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:20.268462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:20.268476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:20.268522 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:20.269108 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.269619 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.270443 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:20.270648 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:20.270715 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:20.273766 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.131:46631: BlockingRecv error: recv got EOF from 127.28.177.131:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:18:21 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:21 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.129@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:21.030802 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.025411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.129:42801)
0808 02:18:21.025591 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:21.025603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:21.025666 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:21.025840 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:21.025841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:21.026056 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:21.026132 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.026135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.027016 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.027018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.027606 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.027609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.027953 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.027954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.028009 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.028531 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:21.028536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:21.030736 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
W20260808 02:18:21.030969 10104 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:21.025539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:42801 (local address 127.28.177.190:42207)
0808 02:18:21.025689 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.025691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.025700 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:18:21.025728 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.025728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.025739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.025810 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.026205 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.026924 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.027682 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.027877 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.027935 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.030871 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.129:42801: BlockingRecv error: recv got EOF from 127.28.177.129:42801 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
Aug 08 02:18:21 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:18:21 dist-test-slave-3h5h krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.177.130@KRBTEST.COM for oryx/127.28.177.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:18:21.146661 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.141229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42207 (local address 127.28.177.130:46033)
0808 02:18:21.141424 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:21.141435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:21.141502 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:21.141669 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:21.141670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:21.141837 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:21.141887 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.141888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.142653 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.142654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.143213 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.143216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.143639 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.143639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.143680 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.144291 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:21.144296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:21.146571 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.177.190:42207: Server oryx/127.28.177.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":112,"threads_started":1}
W20260808 02:18:21.146924 10104 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:21.141306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:46033 (local address 127.28.177.190:42207)
0808 02:18:21.141385 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.141388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.141504 (+ 116us) server_negotiation.cc:407] Connection header received
0808 02:18:21.141560 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.141561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.141574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.141639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.141964 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.142531 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.143307 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.143524 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.143635 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.146820 (+ 3185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.177.130:46033: BlockingRecv error: recv got EOF from 127.28.177.130:46033 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:18:21.179586 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 9444
I20260808 02:18:21.190909 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 9655
I20260808 02:18:21.200081 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 9848
I20260808 02:18:21.209800 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 9336
2026-08-08T02:18:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17810 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T02:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:21Z Disabled control of system clock
I20260808 02:18:21.247290 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:45069
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:18:21.335069 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:21.335273 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:21.335292 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:21.336431 10111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:21.336468 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:21.336480 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:21.336493 10111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:21.336505 10111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:21.337828 10111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:45069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:21.338028 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:21.338264 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:21.344046 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:21.344084 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:21.344208 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:21.344328 10111 server_base.cc:1034] running on GCE node
I20260808 02:18:21.344609 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:21.344942 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:21.346095 10111 hybrid_clock.cc:648] HybridClock initialized: now 1786155501346078 us; error 32 us; skew 500 ppm
I20260808 02:18:21.347494 10111 webserver.cc:460] Webserver started at http://127.28.177.190:38963/ using document root <none> and password file <none>
I20260808 02:18:21.347790 10111 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:21.347872 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:21.348033 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:21.349263 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "8546a78ad7954844822ffa0a4f24fd89"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "1c95751fcd2e8446007e61b1f4d7c66a"
server_key_iv: "3002b20438524d612f2b9211f05c01ee"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.349650 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "8546a78ad7954844822ffa0a4f24fd89"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "1c95751fcd2e8446007e61b1f4d7c66a"
server_key_iv: "3002b20438524d612f2b9211f05c01ee"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.351060 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:18:21.351954 10130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.352221 10111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:21.352331 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "8546a78ad7954844822ffa0a4f24fd89"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "1c95751fcd2e8446007e61b1f4d7c66a"
server_key_iv: "3002b20438524d612f2b9211f05c01ee"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.352423 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:21.362695 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:21.363071 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:21.363237 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:21.367712 10111 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:45069
I20260808 02:18:21.367756 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:45069 every 8 connection(s)
I20260808 02:18:21.368175 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:18:21.369096 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:21.371846 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89: Bootstrap starting.
I20260808 02:18:21.372184 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 10111
I20260808 02:18:21.372298 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:18:21.372511 29382 external_mini_cluster.cc:1428] Setting key 36bf5f35e704ae6c2a544b9bdefdec40
I20260808 02:18:21.372735 10209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:21.373131 10209 log.cc:824] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:21.374047 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89: No bootstrap required, opened a new log
I20260808 02:18:21.375662 10209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8546a78ad7954844822ffa0a4f24fd89" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45069 } }
I20260808 02:18:21.375833 10209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:21.375901 10209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8546a78ad7954844822ffa0a4f24fd89, State: Initialized, Role: FOLLOWER
I20260808 02:18:21.376080 10209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8546a78ad7954844822ffa0a4f24fd89" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45069 } }
I20260808 02:18:21.376153 10209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:18:21.376176 10209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:18:21.376241 10209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:21.377199 10209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8546a78ad7954844822ffa0a4f24fd89" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45069 } }
I20260808 02:18:21.377293 10209 leader_election.cc:302] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8546a78ad7954844822ffa0a4f24fd89; no voters:
I20260808 02:18:21.377530 10209 leader_election.cc:288] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:18:21.377562 10215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:21.377828 10215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [term 1 LEADER]: Becoming Leader. State: Replica: 8546a78ad7954844822ffa0a4f24fd89, State: Running, Role: LEADER
I20260808 02:18:21.378036 10215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8546a78ad7954844822ffa0a4f24fd89" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45069 } }
I20260808 02:18:21.378309 10209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:18:21.378486 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8546a78ad7954844822ffa0a4f24fd89. Latest consensus state: current_term: 1 leader_uuid: "8546a78ad7954844822ffa0a4f24fd89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8546a78ad7954844822ffa0a4f24fd89" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45069 } } }
I20260808 02:18:21.378644 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:21.379055 10218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8546a78ad7954844822ffa0a4f24fd89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8546a78ad7954844822ffa0a4f24fd89" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45069 } } }
I20260808 02:18:21.379158 10218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89 [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:21.379347 10222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:18:21.379412 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.373209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.28.177.190:45069)
0808 02:18:21.373444 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.373454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.373493 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:18:21.374159 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.374198 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.374286 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.374424 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.374684 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.375765 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.376806 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.377472 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.377632 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.378429 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:21.378434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:21.378434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:21.378687 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:21.378732 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:21.378749 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:21.378832 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:21.379053 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 02:18:21.379148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260808 02:18:21.380204 10222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:18:21.382059 10222 catalog_manager.cc:1347] Generated new cluster ID: ca1a52dec49c4fa29e0330e07bfa532f
I20260808 02:18:21.382117 10222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:18:21.389472 10222 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:18:21.390005 10222 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:18:21.395526 10222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8546a78ad7954844822ffa0a4f24fd89: Generated new TSK 0
I20260808 02:18:21.395716 10222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:18:21.399673 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45069
--builtin_ntp_servers=127.28.177.148:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 02:18:21.496594 10247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:21.496806 10247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:21.496882 10247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:21.498081 10247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:21.498131 10247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:21.498158 10247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:18:21.499562 10247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:21.499835 10247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:21.500175 10247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:21.506809 10254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:21.506925 10247 server_base.cc:1034] running on GCE node
W20260808 02:18:21.506839 10253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:21.507030 10256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:21.507277 10247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:21.507576 10247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:21.508751 10247 hybrid_clock.cc:648] HybridClock initialized: now 1786155501508731 us; error 37 us; skew 500 ppm
I20260808 02:18:21.510258 10247 webserver.cc:460] Webserver started at http://127.28.177.129:37965/ using document root <none> and password file <none>
I20260808 02:18:21.510551 10247 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:21.510601 10247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:21.510751 10247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:21.511850 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "20bafc2a8ac74d50bad0cd7d1091c88d"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "8262dcb3264164e61b947e38caa71fd6"
server_key_iv: "7c4684d2ede96f1c4b39194acd1291a2"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.512252 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "20bafc2a8ac74d50bad0cd7d1091c88d"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "8262dcb3264164e61b947e38caa71fd6"
server_key_iv: "7c4684d2ede96f1c4b39194acd1291a2"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.513819 10247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:18:21.514684 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.514881 10247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:21.514977 10247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "20bafc2a8ac74d50bad0cd7d1091c88d"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "8262dcb3264164e61b947e38caa71fd6"
server_key_iv: "7c4684d2ede96f1c4b39194acd1291a2"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.515057 10247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:21.545426 10247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:21.545809 10247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:21.545980 10247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:21.546249 10247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:21.546667 10247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:21.546725 10247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.546770 10247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:21.546797 10247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.552935 10247 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:43145
I20260808 02:18:21.553156 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:43145 every 8 connection(s)
I20260808 02:18:21.553395 10247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:18:21.556958 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 10247
I20260808 02:18:21.557061 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:18:21.557319 29382 external_mini_cluster.cc:1428] Setting key a848f6990c6b4ecc31be5412e08d35fc
I20260808 02:18:21.558457 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45069
--builtin_ntp_servers=127.28.177.148:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:18:21.559993 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.554306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53431 (local address 127.28.177.190:45069)
0808 02:18:21.554393 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.554395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.554619 (+ 224us) server_negotiation.cc:407] Connection header received
0808 02:18:21.555168 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.555169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.555185 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.555267 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.556023 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.556681 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.558190 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.558502 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.558590 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.559639 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:21.559640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:21.559641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:21.559646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:21.559664 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:21.559669 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:21.559731 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:21.559841 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 02:18:21.559872 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:18:21.560227 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.554288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45069 (local address 127.28.177.129:53431)
0808 02:18:21.554489 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:21.554532 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:21.555025 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:21.555322 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:21.555326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:21.555335 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:21.555904 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.555910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.556794 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.556796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.558064 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.558068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.558674 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.558675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.558833 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.559483 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:18:21.559484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:21.559514 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:18:21.559539 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:21.559740 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:21.559745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:21.559747 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:18:21.559803 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:18:21.559942 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260808 02:18:21.560969 10438 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45069
I20260808 02:18:21.561115 10438 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:21.561326 10438 heartbeater.cc:511] Master 127.28.177.190:45069 requested a full tablet report, sending...
I20260808 02:18:21.562000 10155 ts_manager.cc:194] Registered new tserver with Master: 20bafc2a8ac74d50bad0cd7d1091c88d (127.28.177.129:43145)
I20260808 02:18:21.562983 10155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.129:53431
W20260808 02:18:21.643488 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:21.643708 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:21.643730 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:21.644930 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:21.644974 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:21.645000 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:18:21.646363 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:21.646571 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:21.646904 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:21.652390 10452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:21.652390 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:21.652417 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:21.652683 10441 server_base.cc:1034] running on GCE node
I20260808 02:18:21.652988 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:21.653265 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:21.654466 10441 hybrid_clock.cc:648] HybridClock initialized: now 1786155501654458 us; error 40 us; skew 500 ppm
I20260808 02:18:21.656031 10441 webserver.cc:460] Webserver started at http://127.28.177.130:40343/ using document root <none> and password file <none>
I20260808 02:18:21.656347 10441 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:21.656421 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:21.656569 10441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:21.657740 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "7fd9f97df56b49d399ceb40f076b462e"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "60366e10845077092002beb6125d462e"
server_key_iv: "bd445faf2472cf2399c79cf1ec26a9f5"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.658049 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "7fd9f97df56b49d399ceb40f076b462e"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "60366e10845077092002beb6125d462e"
server_key_iv: "bd445faf2472cf2399c79cf1ec26a9f5"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.659348 10441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:18:21.660159 10462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.660373 10441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:21.660454 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "7fd9f97df56b49d399ceb40f076b462e"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "60366e10845077092002beb6125d462e"
server_key_iv: "bd445faf2472cf2399c79cf1ec26a9f5"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.660514 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:21.689406 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:21.689708 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:21.689815 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:21.690016 10441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:21.690438 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:21.690513 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.690557 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:21.690582 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.696784 10441 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:40931
I20260808 02:18:21.696854 10634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:40931 every 8 connection(s)
I20260808 02:18:21.697214 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:18:21.702791 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.697994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:55811 (local address 127.28.177.190:45069)
0808 02:18:21.698052 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.698055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.698297 (+ 242us) server_negotiation.cc:407] Connection header received
0808 02:18:21.698826 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.698828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.698840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.698914 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.699671 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.700219 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.701353 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.701562 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.701631 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.702531 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:21.702532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:21.702532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:21.702537 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:21.702551 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:21.702555 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:21.702605 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:21.702703 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:18:21.702729 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:18:21.702986 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.697917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45069 (local address 127.28.177.130:55811)
0808 02:18:21.698173 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:21.698217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:21.698721 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:21.698991 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:21.698995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:21.699004 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:21.699551 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.699556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.700310 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.700311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.701235 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.701238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.701670 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.701671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.701796 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.702362 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:18:21.702362 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:21.702392 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:18:21.702418 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:21.702615 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:21.702620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:21.702622 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:18:21.702685 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:18:21.702770 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260808 02:18:21.703501 10636 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45069
I20260808 02:18:21.703596 10636 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:21.703744 10636 heartbeater.cc:511] Master 127.28.177.190:45069 requested a full tablet report, sending...
I20260808 02:18:21.704128 10155 ts_manager.cc:194] Registered new tserver with Master: 7fd9f97df56b49d399ceb40f076b462e (127.28.177.130:40931)
I20260808 02:18:21.704569 10155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.130:55811
I20260808 02:18:21.705726 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 10441
I20260808 02:18:21.705806 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:18:21.706022 29382 external_mini_cluster.cc:1428] Setting key 4a1c443aae7a5d230a28949c38776c04
I20260808 02:18:21.706952 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45069
--builtin_ntp_servers=127.28.177.148:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 02:18:21.792650 10639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:21.792888 10639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:21.792912 10639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:21.794106 10639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:21.794155 10639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:21.794179 10639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:18:21.795539 10639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:21.795820 10639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:21.796132 10639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:21.801515 10645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:21.801541 10639 server_base.cc:1034] running on GCE node
W20260808 02:18:21.801523 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:21.801523 10644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:21.802090 10639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:21.802363 10639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:21.803544 10639 hybrid_clock.cc:648] HybridClock initialized: now 1786155501803531 us; error 28 us; skew 500 ppm
I20260808 02:18:21.805102 10639 webserver.cc:460] Webserver started at http://127.28.177.131:34093/ using document root <none> and password file <none>
I20260808 02:18:21.805382 10639 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:21.805459 10639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:21.805611 10639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:21.806728 10639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "acf3dd2c8f2e40fbaf4c8dfeb5c3bbf6"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "5c9283ba8942206ced45434a45651636"
server_key_iv: "6fe57d230cf0be7565eaefe4f14ad1d6"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.807097 10639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "acf3dd2c8f2e40fbaf4c8dfeb5c3bbf6"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "5c9283ba8942206ced45434a45651636"
server_key_iv: "6fe57d230cf0be7565eaefe4f14ad1d6"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.808475 10639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:18:21.809415 10657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.809732 10639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:18:21.809850 10639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "acf3dd2c8f2e40fbaf4c8dfeb5c3bbf6"
format_stamp: "Formatted at 2026-08-08 02:18:21 on dist-test-slave-3h5h"
server_key: "5c9283ba8942206ced45434a45651636"
server_key_iv: "6fe57d230cf0be7565eaefe4f14ad1d6"
server_key_version: "encryptionkey@0"
I20260808 02:18:21.809942 10639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:21.823578 10639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:21.823952 10639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:21.824085 10639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:21.824357 10639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:21.824730 10639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:21.824795 10639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.824872 10639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:21.824906 10639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:21.831038 10639 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:41991
I20260808 02:18:21.831084 10821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:41991 every 8 connection(s)
I20260808 02:18:21.831516 10639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:18:21.833168 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 10639
I20260808 02:18:21.833271 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:18:21.833498 29382 external_mini_cluster.cc:1428] Setting key 76b8a990a3680a46c76f69606f4f3c1c
I20260808 02:18:21.842015 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.832802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:45883 (local address 127.28.177.190:45069)
0808 02:18:21.832922 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.832924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.833173 (+ 249us) server_negotiation.cc:407] Connection header received
0808 02:18:21.833930 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.833931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.833944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.834001 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.834474 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.835297 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.840341 (+ 5044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.840545 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.840639 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.841682 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:21.841684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:21.841684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:21.841690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:21.841706 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:21.841715 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:21.841774 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:21.841894 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 02:18:21.841925 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:18:21.842301 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:21.832728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45069 (local address 127.28.177.131:45883)
0808 02:18:21.833002 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:21.833052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:21.833792 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:21.834081 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:21.834085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:21.834095 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:21.834398 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.834404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.839254 (+ 4850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.839255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.840209 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:21.840212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:21.840641 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:21.840642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:21.840802 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:21.841491 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:18:21.841491 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:21.841524 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:18:21.841551 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:21.841793 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:21.841800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:21.841803 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:21.841875 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:18:21.842011 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":139,"threads_started":1}
I20260808 02:18:21.842818 10822 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45069
I20260808 02:18:21.842954 10822 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:21.843220 10822 heartbeater.cc:511] Master 127.28.177.190:45069 requested a full tablet report, sending...
I20260808 02:18:21.843679 10155 ts_manager.cc:194] Registered new tserver with Master: acf3dd2c8f2e40fbaf4c8dfeb5c3bbf6 (127.28.177.131:41991)
I20260808 02:18:21.844060 10155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.131:45883
I20260808 02:18:21.845544 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:18:21.849478 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:21.847016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.28.177.190:45069)
0808 02:18:21.847086 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:21.847088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:21.849059 (+ 1971us) server_negotiation.cc:407] Connection header received
0808 02:18:21.849132 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:21.849133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:21.849143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:21.849197 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:21.849386 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35008: BlockingRecv error: recv got EOF from 127.0.0.1:35008 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:18:21.849572 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 10247
I20260808 02:18:21.874013 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 10441
I20260808 02:18:21.884619 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 10639
I20260808 02:18:21.900502 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 10111
2026-08-08T02:18:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (685 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T02:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:21Z Disabled control of system clock
I20260808 02:18:21.932577 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36299
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:18:22.017599 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:22.017887 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:22.017930 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:22.019759 10835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:22.019840 10835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:22.019863 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:22.019884 10835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:22.019917 10835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:22.021401 10835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:36299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:22.021721 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:22.022051 10835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:22.027489 10841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:22.027491 10840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:22.027518 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:22.027846 10835 server_base.cc:1034] running on GCE node
I20260808 02:18:22.028158 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:22.028470 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:22.029618 10835 hybrid_clock.cc:648] HybridClock initialized: now 1786155502029595 us; error 39 us; skew 500 ppm
I20260808 02:18:22.030969 10835 webserver.cc:460] Webserver started at http://127.28.177.190:37649/ using document root <none> and password file <none>
I20260808 02:18:22.031265 10835 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:22.031343 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:22.031500 10835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:22.032608 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "cbe55127a7b646e6b7bad26d8f97178d"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "5d32bbddc4560735f6c64c9b9abc784b"
server_key_iv: "f2abb98b3ee72f51050ca79744f1a20d"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.033002 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "cbe55127a7b646e6b7bad26d8f97178d"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "5d32bbddc4560735f6c64c9b9abc784b"
server_key_iv: "f2abb98b3ee72f51050ca79744f1a20d"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.034468 10835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:22.035413 10852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.035666 10835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:22.035779 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "cbe55127a7b646e6b7bad26d8f97178d"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "5d32bbddc4560735f6c64c9b9abc784b"
server_key_iv: "f2abb98b3ee72f51050ca79744f1a20d"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.035859 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:22.039351 10835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:22.039619 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:22.039768 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:22.043869 10835 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:36299
I20260808 02:18:22.043903 10925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:36299 every 8 connection(s)
I20260808 02:18:22.044250 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:18:22.045024 10927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:22.047500 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 10835
I20260808 02:18:22.047593 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:18:22.047807 29382 external_mini_cluster.cc:1428] Setting key 771891f7ee7c2d1fdcec66b1b0965261
I20260808 02:18:22.048167 10927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d: Bootstrap starting.
I20260808 02:18:22.048923 10927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:22.049285 10927 log.cc:824] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:22.050479 10927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d: No bootstrap required, opened a new log
I20260808 02:18:22.050761 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.048408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.28.177.190:36299)
0808 02:18:22.048640 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:22.048649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:22.048683 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:18:22.049358 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:22.049368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:22.049422 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:22.049590 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:22.050293 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:22.050297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:22.050297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:22.050302 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:22.050351 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:22.050365 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:22.050459 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:22.050491 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 02:18:22.050559 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260808 02:18:22.052677 10927 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe55127a7b646e6b7bad26d8f97178d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36299 } }
I20260808 02:18:22.052907 10927 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:22.052958 10927 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe55127a7b646e6b7bad26d8f97178d, State: Initialized, Role: FOLLOWER
I20260808 02:18:22.053117 10927 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe55127a7b646e6b7bad26d8f97178d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36299 } }
I20260808 02:18:22.053210 10927 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:18:22.053254 10927 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:18:22.053304 10927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:22.053880 10927 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe55127a7b646e6b7bad26d8f97178d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36299 } }
I20260808 02:18:22.054000 10927 leader_election.cc:302] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe55127a7b646e6b7bad26d8f97178d; no voters:
I20260808 02:18:22.054180 10927 leader_election.cc:288] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:18:22.054260 10936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:22.054420 10936 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [term 1 LEADER]: Becoming Leader. State: Replica: cbe55127a7b646e6b7bad26d8f97178d, State: Running, Role: LEADER
I20260808 02:18:22.054484 10927 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:18:22.054577 10936 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe55127a7b646e6b7bad26d8f97178d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36299 } }
I20260808 02:18:22.055042 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe55127a7b646e6b7bad26d8f97178d. Latest consensus state: current_term: 1 leader_uuid: "cbe55127a7b646e6b7bad26d8f97178d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe55127a7b646e6b7bad26d8f97178d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36299 } } }
I20260808 02:18:22.055121 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:22.055016 10937 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe55127a7b646e6b7bad26d8f97178d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe55127a7b646e6b7bad26d8f97178d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 36299 } } }
I20260808 02:18:22.055176 10937 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:22.055452 10951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:18:22.055960 10951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:18:22.057554 10951 catalog_manager.cc:1347] Generated new cluster ID: 1e5bc6f6294b4b03a1fa1edb9f75078f
I20260808 02:18:22.057614 10951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:18:22.061502 10951 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:18:22.061668 10951 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:18:22.070228 10951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe55127a7b646e6b7bad26d8f97178d: Generated new TSK 0
I20260808 02:18:22.070503 10951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:18:22.073109 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36299
--builtin_ntp_servers=127.28.177.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:18:22.176424 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:22.176630 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:22.176654 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:22.177883 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:22.177932 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:22.177958 10969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:18:22.179437 10969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:22.179721 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:22.180052 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:22.185670 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:22.185731 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:22.185864 10969 server_base.cc:1034] running on GCE node
W20260808 02:18:22.186057 10979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:22.186324 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:22.186648 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:22.187844 10969 hybrid_clock.cc:648] HybridClock initialized: now 1786155502187821 us; error 37 us; skew 500 ppm
I20260808 02:18:22.189452 10969 webserver.cc:460] Webserver started at http://127.28.177.129:40215/ using document root <none> and password file <none>
I20260808 02:18:22.189757 10969 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:22.189836 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:22.189978 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:22.191478 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "0764178ce6a24f368d253f5e3b964daf"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "c0c399ea107fb7fc4a5a44bb402f4acc"
server_key_iv: "f8b3e0e05be58c6aa5fc6b9203e0e21f"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.191921 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "0764178ce6a24f368d253f5e3b964daf"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "c0c399ea107fb7fc4a5a44bb402f4acc"
server_key_iv: "f8b3e0e05be58c6aa5fc6b9203e0e21f"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.193725 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260808 02:18:22.194736 10990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.194963 10969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:22.195068 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "0764178ce6a24f368d253f5e3b964daf"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "c0c399ea107fb7fc4a5a44bb402f4acc"
server_key_iv: "f8b3e0e05be58c6aa5fc6b9203e0e21f"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.195158 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:22.200552 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:22.200907 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:22.201066 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:22.201316 10969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:22.201682 10969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:22.201733 10969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.201763 10969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:22.201781 10969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.209031 10969 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:43229
I20260808 02:18:22.209229 11145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:43229 every 8 connection(s)
I20260808 02:18:22.209533 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:18:22.210246 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 10969
I20260808 02:18:22.210364 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:18:22.210597 29382 external_mini_cluster.cc:1428] Setting key eae9b3c03a559dd660706e916a0560e6
I20260808 02:18:22.211777 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36299
--builtin_ntp_servers=127.28.177.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:18:22.214044 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.211198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:41221 (local address 127.28.177.190:36299)
0808 02:18:22.211292 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:22.211295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:22.211537 (+ 242us) server_negotiation.cc:407] Connection header received
0808 02:18:22.212173 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:22.212175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:22.212182 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:22.212251 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:22.213754 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:22.213755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:22.213755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:22.213761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:22.213777 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:22.213781 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:22.213813 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:22.213955 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 02:18:22.213970 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:18:22.214476 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.211191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36299 (local address 127.28.177.129:41221)
0808 02:18:22.211414 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:22.211460 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:22.212053 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:22.212325 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:22.212330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:22.212340 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:22.213544 (+ 1204us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:18:22.213545 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:22.213588 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:18:22.213624 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:22.213864 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:22.213872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:22.213875 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:22.214195 (+ 320us) client_negotiation.cc:240] Negotiation successful
0808 02:18:22.214263 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260808 02:18:22.215276 11146 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36299
I20260808 02:18:22.215399 11146 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:22.215584 11146 heartbeater.cc:511] Master 127.28.177.190:36299 requested a full tablet report, sending...
I20260808 02:18:22.216257 10875 ts_manager.cc:194] Registered new tserver with Master: 0764178ce6a24f368d253f5e3b964daf (127.28.177.129:43229)
W20260808 02:18:22.307700 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:22.307919 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:22.307943 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:22.309198 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:22.309242 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:22.309269 11150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:18:22.310704 11150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:22.310976 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:22.311290 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:22.317078 11158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:22.317082 11162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:22.317098 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:22.317682 11150 server_base.cc:1034] running on GCE node
I20260808 02:18:22.317971 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:22.318261 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:22.319425 11150 hybrid_clock.cc:648] HybridClock initialized: now 1786155502319410 us; error 30 us; skew 500 ppm
I20260808 02:18:22.320981 11150 webserver.cc:460] Webserver started at http://127.28.177.130:36557/ using document root <none> and password file <none>
I20260808 02:18:22.321275 11150 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:22.321408 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:22.321592 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:22.322793 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "398557372c8c4d9cb11b80550fffd12c"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "cadab5ba7a01c9fe0e1d9e85af1092a1"
server_key_iv: "056299ab664efc1d50cb5249f60c0ccd"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.323236 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "398557372c8c4d9cb11b80550fffd12c"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "cadab5ba7a01c9fe0e1d9e85af1092a1"
server_key_iv: "056299ab664efc1d50cb5249f60c0ccd"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.324805 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:18:22.325821 11172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.326079 11150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:22.326213 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "398557372c8c4d9cb11b80550fffd12c"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "cadab5ba7a01c9fe0e1d9e85af1092a1"
server_key_iv: "056299ab664efc1d50cb5249f60c0ccd"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.326303 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:22.330156 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:22.330457 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:22.330616 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:22.330878 11150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:22.331214 11150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:22.331280 11150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.331326 11150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:22.331382 11150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.338267 11150 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:38477
I20260808 02:18:22.338351 11335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:38477 every 8 connection(s)
I20260808 02:18:22.338717 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:18:22.340274 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 11150
I20260808 02:18:22.340404 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:18:22.340647 29382 external_mini_cluster.cc:1428] Setting key e0f09f90502be3d42437b4af853ab88b
I20260808 02:18:22.341782 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36299
--builtin_ntp_servers=127.28.177.148:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:18:22.342501 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.339642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:45703 (local address 127.28.177.190:36299)
0808 02:18:22.339710 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:22.339713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:22.340016 (+ 303us) server_negotiation.cc:407] Connection header received
0808 02:18:22.340706 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:22.340707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:22.340715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:22.340780 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:22.342194 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:22.342195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:22.342196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:22.342201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:22.342218 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:22.342223 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:22.342291 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:22.342402 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:18:22.342420 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:18:22.342724 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.339661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36299 (local address 127.28.177.130:45703)
0808 02:18:22.339883 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:22.339919 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:22.340577 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:22.340883 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:22.340888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:22.340898 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:22.342011 (+ 1113us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:18:22.342011 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:22.342042 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:18:22.342074 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:22.342294 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:22.342300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:22.342304 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:22.342414 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:18:22.342484 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260808 02:18:22.343201 11336 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36299
I20260808 02:18:22.343331 11336 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:22.343495 11336 heartbeater.cc:511] Master 127.28.177.190:36299 requested a full tablet report, sending...
I20260808 02:18:22.343942 10875 ts_manager.cc:194] Registered new tserver with Master: 398557372c8c4d9cb11b80550fffd12c (127.28.177.130:38477)
W20260808 02:18:22.433189 11340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:22.433437 11340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:22.433476 11340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:22.434674 11340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:22.434748 11340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:22.434795 11340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:18:22.436317 11340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:22.436589 11340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:22.436945 11340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:22.442628 11351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:22.442674 11346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:22.442741 11340 server_base.cc:1034] running on GCE node
W20260808 02:18:22.442675 11347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:22.443118 11340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:22.443463 11340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:22.444656 11340 hybrid_clock.cc:648] HybridClock initialized: now 1786155502444627 us; error 47 us; skew 500 ppm
I20260808 02:18:22.446013 11340 webserver.cc:460] Webserver started at http://127.28.177.131:38009/ using document root <none> and password file <none>
I20260808 02:18:22.446359 11340 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:22.446441 11340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:22.446589 11340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:22.447693 11340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "5e0af05247a4479d8b42e4c01e98b03d"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "15ba238be2cf576c1438f61cef7ac708"
server_key_iv: "558e3dd1700eb612c8c6be5959f673ba"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.448084 11340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "5e0af05247a4479d8b42e4c01e98b03d"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "15ba238be2cf576c1438f61cef7ac708"
server_key_iv: "558e3dd1700eb612c8c6be5959f673ba"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.449493 11340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 02:18:22.450317 11359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.450501 11340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:22.450608 11340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "5e0af05247a4479d8b42e4c01e98b03d"
format_stamp: "Formatted at 2026-08-08 02:18:22 on dist-test-slave-3h5h"
server_key: "15ba238be2cf576c1438f61cef7ac708"
server_key_iv: "558e3dd1700eb612c8c6be5959f673ba"
server_key_version: "encryptionkey@0"
I20260808 02:18:22.450696 11340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:22.454325 11340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:22.454610 11340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:22.454759 11340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:22.455039 11340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:22.455399 11340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:22.455433 11340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.455454 11340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:22.455467 11340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:22.461699 11340 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:41453
I20260808 02:18:22.461761 11518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:41453 every 8 connection(s)
I20260808 02:18:22.462085 11340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:18:22.465476 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.463025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:35055 (local address 127.28.177.190:36299)
0808 02:18:22.463134 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:22.463136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:22.463353 (+ 217us) server_negotiation.cc:407] Connection header received
0808 02:18:22.463840 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:22.463842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:22.463848 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:22.463914 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:22.465255 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:22.465257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:22.465257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:18:22.465261 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:22.465278 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:18:22.465283 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:22.465345 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:22.465402 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 02:18:22.465416 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:18:22.465687 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:22.462995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:36299 (local address 127.28.177.131:35055)
0808 02:18:22.463195 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:22.463237 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:22.463747 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:22.463970 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:22.463974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:22.463982 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:22.465080 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:18:22.465081 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:22.465112 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:18:22.465147 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:22.465346 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:22.465348 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:22.465351 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:18:22.465399 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 02:18:22.465453 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260808 02:18:22.466243 11519 heartbeater.cc:347] Connected to a master server at 127.28.177.190:36299
I20260808 02:18:22.466331 11519 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:22.466454 11519 heartbeater.cc:511] Master 127.28.177.190:36299 requested a full tablet report, sending...
I20260808 02:18:22.466828 10875 ts_manager.cc:194] Registered new tserver with Master: 5e0af05247a4479d8b42e4c01e98b03d (127.28.177.131:41453)
I20260808 02:18:22.468310 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 11340
I20260808 02:18:22.468396 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:18:22.468580 29382 external_mini_cluster.cc:1428] Setting key 3f9009a1c8e57d463e12dc36c550ed22
I20260808 02:18:22.469681 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:18:22.471746 10930 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:18:22.471110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.28.177.190:36299)
0808 02:18:22.471188 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:22.471190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:22.471313 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:18:22.471398 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:22.471399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:22.471405 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:22.471463 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:22.471614 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41624: BlockingRecv error: recv got EOF from 127.0.0.1:41624 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:18:22.471833 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 10969
I20260808 02:18:22.485932 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 11150
I20260808 02:18:22.499727 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 11340
I20260808 02:18:22.508245 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 10835
2026-08-08T02:18:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (614 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:18:22 dist-test-slave-3h5h krb5kdc[11532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:18:22 dist-test-slave-3h5h krb5kdc[11532](info): set up 2 sockets
Aug 08 02:18:22 dist-test-slave-3h5h krb5kdc[11532](info): commencing operation
krb5kdc: starting...
W20260808 02:18:23.766147 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:18:23 dist-test-slave-3h5h krb5kdc[11532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:18:23.916158 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:43217
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:24.007309 11551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:24.007537 11551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:24.007560 11551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:24.008800 11551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:18:24.008908 11551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:24.008946 11551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:24.008960 11551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:18:24.009016 11551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:18:24.010756 11551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:43217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:24.011072 11551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:24.011334 11551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:24.017187 11560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:24.017252 11551 server_base.cc:1034] running on GCE node
W20260808 02:18:24.017187 11559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:24.017187 11562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:24.017745 11551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:24.018026 11551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:24.019232 11551 hybrid_clock.cc:648] HybridClock initialized: now 1786155504019212 us; error 42 us; skew 500 ppm
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:24.022763 11551 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:18:24.023264 11551 webserver.cc:460] Webserver started at http://127.28.177.190:39161/ using document root <none> and password file <none>
I20260808 02:18:24.023519 11551 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:24.023573 11551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:24.023682 11551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:24.024919 11551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "d3f8eaef9d4141a8988fed4912457034"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "740b1b735e47a2dc600e86650b4de97c"
server_key_iv: "b1fc983d710d2bbf4cb7738943c740df"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.025295 11551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "d3f8eaef9d4141a8988fed4912457034"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "740b1b735e47a2dc600e86650b4de97c"
server_key_iv: "b1fc983d710d2bbf4cb7738943c740df"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.026865 11551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:18:24.027838 11571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.028082 11551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:24.028168 11551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "d3f8eaef9d4141a8988fed4912457034"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "740b1b735e47a2dc600e86650b4de97c"
server_key_iv: "b1fc983d710d2bbf4cb7738943c740df"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.028234 11551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:24.046305 11551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:24.048789 11551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:24.049026 11551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:24.053320 11551 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:43217
I20260808 02:18:24.053397 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:43217 every 8 connection(s)
I20260808 02:18:24.053754 11551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:18:24.055066 11646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:24.057873 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: Bootstrap starting.
I20260808 02:18:24.058570 11646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:24.058899 11646 log.cc:824] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:24.059849 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: No bootstrap required, opened a new log
I20260808 02:18:24.061956 11646 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f8eaef9d4141a8988fed4912457034" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43217 } }
I20260808 02:18:24.062152 11646 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:24.062207 11646 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f8eaef9d4141a8988fed4912457034, State: Initialized, Role: FOLLOWER
I20260808 02:18:24.062376 11646 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f8eaef9d4141a8988fed4912457034" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43217 } }
I20260808 02:18:24.062485 11646 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:18:24.062537 11646 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:18:24.062592 11646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:24.063298 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 11551
I20260808 02:18:24.063441 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:18:24.063652 29382 external_mini_cluster.cc:1428] Setting key 5e213159746d88f64a24ac4f2167c356
I20260808 02:18:24.063812 11646 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f8eaef9d4141a8988fed4912457034" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43217 } }
I20260808 02:18:24.063934 11646 leader_election.cc:302] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f8eaef9d4141a8988fed4912457034; no voters:
I20260808 02:18:24.064174 11652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:24.064332 11652 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [term 1 LEADER]: Becoming Leader. State: Replica: d3f8eaef9d4141a8988fed4912457034, State: Running, Role: LEADER
I20260808 02:18:24.064378 11646 leader_election.cc:288] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:18:24.064716 11646 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:18:24.065215 11652 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f8eaef9d4141a8988fed4912457034" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43217 } }
I20260808 02:18:24.066644 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f8eaef9d4141a8988fed4912457034. Latest consensus state: current_term: 1 leader_uuid: "d3f8eaef9d4141a8988fed4912457034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f8eaef9d4141a8988fed4912457034" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43217 } } }
I20260808 02:18:24.066756 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [sys.catalog]: This master's current role is: LEADER
W20260808 02:18:24.067395 11681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:18:24.067499 11681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:18:24.067559 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3f8eaef9d4141a8988fed4912457034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f8eaef9d4141a8988fed4912457034" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43217 } } }
I20260808 02:18:24.067622 11684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:18:24.067672 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034 [sys.catalog]: This master's current role is: LEADER
I20260808 02:18:24.067819 11684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:18:24.069432 11684 catalog_manager.cc:1347] Generated new cluster ID: dc98d26f1f4a4dbcb2d9d5c650b3f847
I20260808 02:18:24.069492 11684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:24.076574 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.064218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.28.177.190:43217)
0808 02:18:24.064483 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.064495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.064541 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:18:24.064714 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.064724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.064791 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.065004 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:24.065833 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.067336 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.068431 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.069185 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.069308 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.072388 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:24.072393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:24.072394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:24.072407 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:24.074574 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.075052 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.075055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.075056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.075186 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.075443 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.075444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.075444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.075766 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:24.075908 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.076079 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.076220 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"spinlock_wait_cycles":11136,"thread_start_us":103,"threads_started":1}
I20260808 02:18:24.086287 11684 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:18:24.086836 11684 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:18:24.095721 11684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: Generated new TSK 0
I20260808 02:18:24.096061 11684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:18:24.132517 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:43217
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:24.225349 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:24.225631 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:24.225685 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:24.227042 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:24.227121 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:24.227169 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:18:24.229334 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:24.229677 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:24.229967 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:24.237205 11698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:24.237450 11690 server_base.cc:1034] running on GCE node
W20260808 02:18:24.237205 11700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:24.237275 11702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:24.237841 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:24.238147 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:24.239362 11690 hybrid_clock.cc:648] HybridClock initialized: now 1786155504239332 us; error 48 us; skew 500 ppm
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:24.242476 11690 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:18:24.242971 11690 webserver.cc:460] Webserver started at http://127.28.177.129:34845/ using document root <none> and password file <none>
I20260808 02:18:24.243322 11690 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:24.243409 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:24.243577 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:24.244694 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "cad3d3ac18bc4099bbc810ad172fe000"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "75659aca2c32cc1b45ac9791f44db2a3"
server_key_iv: "0cf766ea06f21feeaa93cd5f9f9d3f27"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.245121 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "cad3d3ac18bc4099bbc810ad172fe000"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "75659aca2c32cc1b45ac9791f44db2a3"
server_key_iv: "0cf766ea06f21feeaa93cd5f9f9d3f27"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.246806 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:24.247822 11710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.248317 11690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:24.248451 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "cad3d3ac18bc4099bbc810ad172fe000"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "75659aca2c32cc1b45ac9791f44db2a3"
server_key_iv: "0cf766ea06f21feeaa93cd5f9f9d3f27"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.248558 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:24.257093 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:24.259457 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:24.259747 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:24.260002 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:24.260351 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:24.260414 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.260458 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:24.260481 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.267457 11690 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:38043
I20260808 02:18:24.267517 11880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:38043 every 8 connection(s)
I20260808 02:18:24.267870 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:18:24.269080 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 11690
I20260808 02:18:24.269191 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:18:24.269403 29382 external_mini_cluster.cc:1428] Setting key 5f4fb0e00618e6316f86bdbbde679889
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:18:24.280054 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.269047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:39277 (local address 127.28.177.190:43217)
0808 02:18:24.269620 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.269623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.269637 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:18:24.270064 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.270065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.270087 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.270183 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:24.271584 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.272314 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.273800 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.274007 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.274131 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.276954 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:24.276955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:24.276956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:24.276971 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:24.278475 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.278997 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.278998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.278998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.279057 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.279336 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.279337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.279338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.279472 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:24.279557 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.279863 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.279957 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544}
I20260808 02:18:24.280263 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.268918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:43217 (local address 127.28.177.129:39277)
0808 02:18:24.269152 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:24.269212 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:24.269916 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:24.270267 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:24.270270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:24.271174 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:24.271458 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.271463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.272487 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.272489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.273702 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.273706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.274165 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.274167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.274346 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.274966 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:24.274980 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:24.276777 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:24.278616 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:24.278625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:24.278627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:24.278876 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:24.279137 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:24.279138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:24.279138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:24.279256 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:24.279554 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:24.279563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:24.279715 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 02:18:24.279824 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:18:24.279996 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":116,"threads_started":1}
I20260808 02:18:24.281095 11881 heartbeater.cc:347] Connected to a master server at 127.28.177.190:43217
I20260808 02:18:24.281224 11881 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:24.281471 11881 heartbeater.cc:511] Master 127.28.177.190:43217 requested a full tablet report, sending...
I20260808 02:18:24.282105 11601 ts_manager.cc:194] Registered new tserver with Master: cad3d3ac18bc4099bbc810ad172fe000 (127.28.177.129:38043)
I20260808 02:18:24.282770 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:39277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:18:24.300137 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:43217
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:24.391758 11888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:24.392086 11888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:24.392153 11888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:24.393497 11888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:24.393582 11888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:24.393636 11888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:18:24.395576 11888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:24.395900 11888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:24.396273 11888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:18:24.402006 11888 server_base.cc:1034] running on GCE node
W20260808 02:18:24.401988 11899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:24.401983 11895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:24.401971 11894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:24.402472 11888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:24.402767 11888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:24.403950 11888 hybrid_clock.cc:648] HybridClock initialized: now 1786155504403934 us; error 47 us; skew 500 ppm
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:24.406780 11888 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:18:24.407315 11888 webserver.cc:460] Webserver started at http://127.28.177.130:39143/ using document root <none> and password file <none>
I20260808 02:18:24.407557 11888 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:24.407627 11888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:24.407780 11888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:24.408941 11888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "91a77976fe394189a86073d053138a4e"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "0dc361ee544ccdf497d811217d1ede88"
server_key_iv: "455bd070b9c795cfb3eb43cc30a23dd2"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.409337 11888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "91a77976fe394189a86073d053138a4e"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "0dc361ee544ccdf497d811217d1ede88"
server_key_iv: "455bd070b9c795cfb3eb43cc30a23dd2"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.410704 11888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:18:24.411547 11910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.411731 11888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:18:24.411837 11888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "91a77976fe394189a86073d053138a4e"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "0dc361ee544ccdf497d811217d1ede88"
server_key_iv: "455bd070b9c795cfb3eb43cc30a23dd2"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.411921 11888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:24.422297 11888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:24.424399 11888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:24.424589 11888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:24.424873 11888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:24.425220 11888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:24.425276 11888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.425323 11888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:24.425348 11888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.431752 11888 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:34945
I20260808 02:18:24.431823 12076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:34945 every 8 connection(s)
I20260808 02:18:24.432160 11888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:18:24.435997 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 11888
I20260808 02:18:24.436264 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:18:24.436563 29382 external_mini_cluster.cc:1428] Setting key 27e94bc47e66e7debdf23b0b5734f4a2
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:24.443897 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.433099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:50315 (local address 127.28.177.190:43217)
0808 02:18:24.433184 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.433186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.433366 (+ 180us) server_negotiation.cc:407] Connection header received
0808 02:18:24.434092 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.434093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.434107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.434210 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:24.435310 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.435888 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.437485 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.437687 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.437779 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.440429 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:24.440431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:24.440432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:24.440442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:24.442088 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.442592 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.442593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.442593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.442649 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.442944 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.442944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.442944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.443049 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:24.443101 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.443509 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.443724 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:18:24.443977 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.432954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:43217 (local address 127.28.177.130:50315)
0808 02:18:24.433200 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:24.433252 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:24.433871 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:24.434325 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:24.434328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:24.434979 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:24.435194 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.435199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.436067 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.436108 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.437368 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.437372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.437789 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.437790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.437928 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.438521 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:24.438529 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:24.440307 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:24.442158 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:24.442166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:24.442169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:24.442498 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:24.442720 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:24.442721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:24.442722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:24.442875 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:24.443118 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:24.443126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:24.443342 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 02:18:24.443470 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:18:24.443667 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
I20260808 02:18:24.444861 12077 heartbeater.cc:347] Connected to a master server at 127.28.177.190:43217
I20260808 02:18:24.444983 12077 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:24.445205 12077 heartbeater.cc:511] Master 127.28.177.190:43217 requested a full tablet report, sending...
I20260808 02:18:24.445644 11601 ts_manager.cc:194] Registered new tserver with Master: 91a77976fe394189a86073d053138a4e (127.28.177.130:34945)
I20260808 02:18:24.446038 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:50315
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:18:24.468725 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:43217
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:18:24.559616 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:18:24.559819 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:18:24.559844 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:18:24.561009 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:18:24.561051 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:18:24.561079 12084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:18:24.562500 12084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:18:24.562682 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:18:24.562886 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:18:24.568646 12092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:24.568668 12091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:18:24.568702 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:18:24.569001 12084 server_base.cc:1034] running on GCE node
I20260808 02:18:24.569212 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:18:24.569509 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:18:24.570667 12084 hybrid_clock.cc:648] HybridClock initialized: now 1786155504570653 us; error 27 us; skew 500 ppm
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:18:24.573256 12084 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:18:24.573731 12084 webserver.cc:460] Webserver started at http://127.28.177.131:38749/ using document root <none> and password file <none>
I20260808 02:18:24.573992 12084 fs_manager.cc:359] Metadata directory not provided
I20260808 02:18:24.574061 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:18:24.574218 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:18:24.575323 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "67ce19437f51493c821e44eae8e3dba0"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "a4d1ccde68452e922b0102340d1cc694"
server_key_iv: "68a9971e5719c61769179e060f0b52a6"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.575711 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "67ce19437f51493c821e44eae8e3dba0"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "a4d1ccde68452e922b0102340d1cc694"
server_key_iv: "68a9971e5719c61769179e060f0b52a6"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.577205 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:18:24.578061 12104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.578246 12084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:18:24.578341 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "67ce19437f51493c821e44eae8e3dba0"
format_stamp: "Formatted at 2026-08-08 02:18:24 on dist-test-slave-3h5h"
server_key: "a4d1ccde68452e922b0102340d1cc694"
server_key_iv: "68a9971e5719c61769179e060f0b52a6"
server_key_version: "encryptionkey@0"
I20260808 02:18:24.578423 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:18:24.587275 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:18:24.589340 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:18:24.589509 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:18:24.589758 12084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:18:24.590076 12084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:18:24.590133 12084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.590200 12084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:18:24.590231 12084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:18:24.596401 12084 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:45101
I20260808 02:18:24.596441 12261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:45101 every 8 connection(s)
I20260808 02:18:24.596872 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:18:24 dist-test-slave-3h5h krb5kdc[11532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155504, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:18:24.604799 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 12084
I20260808 02:18:24.604964 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:18:24.605176 29382 external_mini_cluster.cc:1428] Setting key 8efbe6f4426f04b8012b281e2736ecbe
I20260808 02:18:24.607060 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.597805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:36025 (local address 127.28.177.190:43217)
0808 02:18:24.597948 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.597951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.598161 (+ 210us) server_negotiation.cc:407] Connection header received
0808 02:18:24.598687 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.598688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.598699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.598753 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:24.599698 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.600221 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.601348 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.601586 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.601662 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.603851 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:24.603853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:24.603853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:24.603860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:24.605568 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.606005 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.606007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.606007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.606079 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.606355 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.606356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.606356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.606485 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:24.606563 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.606870 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.606965 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 02:18:24.607283 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.597769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:43217 (local address 127.28.177.131:36025)
0808 02:18:24.598003 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:24.598054 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:24.598590 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:24.598806 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:24.598810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:24.599388 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:18:24.599585 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.599590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.600315 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.600316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.601247 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.601250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.601688 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.601689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.601792 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.602271 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:18:24.602279 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:18:24.603760 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:18:24.605656 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:24.605664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:24.605667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:24.605905 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:24.606152 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:18:24.606153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:18:24.606153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:18:24.606268 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:18:24.606555 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:18:24.606562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:18:24.606713 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 02:18:24.606823 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:18:24.606996 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260808 02:18:24.607831 12262 heartbeater.cc:347] Connected to a master server at 127.28.177.190:43217
I20260808 02:18:24.608081 12262 heartbeater.cc:464] Registering TS with master...
I20260808 02:18:24.608309 12262 heartbeater.cc:511] Master 127.28.177.190:43217 requested a full tablet report, sending...
I20260808 02:18:24.608733 11601 ts_manager.cc:194] Registered new tserver with Master: 67ce19437f51493c821e44eae8e3dba0 (127.28.177.131:45101)
I20260808 02:18:24.609165 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:36025
I20260808 02:18:24.617357 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:18:24.624912 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.618735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.28.177.190:43217)
0808 02:18:24.618802 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.618804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.618859 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:18:24.618912 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.618912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.618922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.618982 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:18:24.619527 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.620009 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.620770 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.621024 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.621143 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.622320 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:18:24.622321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:18:24.622321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:18:24.622328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:18:24.623784 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.624195 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.624196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.624196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.624251 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:18:24.624409 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:18:24.624410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:18:24.624410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:18:24.624545 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:18:24.624617 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.624706 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.624767 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:18:24.626438 11601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:18:24.628041 11601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:18:24.635541 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.630773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:45101 (local address 127.0.0.1:47728)
0808 02:18:24.631352 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:24.631395 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:24.631484 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:24.632094 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:24.632095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:24.632097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:24.632136 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.632137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.633098 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.633099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.634550 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.634551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.635228 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.635229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.635310 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.635324 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:18:24.635392 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:18:24.635450 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":66,"thread_start_us":61,"threads_started":1}
I20260808 02:18:24.635638 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.630862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.28.177.130:34945)
0808 02:18:24.631270 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.631275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.631484 (+ 209us) server_negotiation.cc:407] Connection header received
0808 02:18:24.631628 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.631631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.631648 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.631741 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:24.631933 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.633086 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.634620 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.635295 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.635357 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.635401 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.635504 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.635548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":96,"threads_started":1}
I20260808 02:18:24.635645 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.630982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:38043 (local address 127.0.0.1:34212)
0808 02:18:24.631354 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:24.631396 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:24.631476 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:24.631761 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:24.631764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:24.631769 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:24.631867 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.631868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.633349 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.633351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.634388 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.634391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.635391 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.635393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.635454 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.635459 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:18:24.635521 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:18:24.635559 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":9,"thread_start_us":80,"threads_started":1}
I20260808 02:18:24.635654 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.630867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:34945 (local address 127.0.0.1:39286)
0808 02:18:24.631353 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:24.631396 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:24.631476 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:24.631761 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:24.631764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:24.631766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:24.631866 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.631868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.633175 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.633176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.634526 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:24.634530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.635369 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.635370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:24.635437 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.635441 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:24.635490 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:18:24.635525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":70,"thread_start_us":84,"threads_started":1}
I20260808 02:18:24.635965 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.630992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.28.177.131:45101)
0808 02:18:24.631895 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.631899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.631909 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:18:24.631947 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.631949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.631965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.632054 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:24.632205 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.632989 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.634641 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.635136 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.635229 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.635289 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.635805 (+ 516us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.635849 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":144,"threads_started":1}
I20260808 02:18:24.636019 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:24.631136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.28.177.129:38043)
0808 02:18:24.631351 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:24.631355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:24.631525 (+ 170us) server_negotiation.cc:407] Connection header received
0808 02:18:24.631578 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:24.631581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:24.631599 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:24.631674 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:24.632165 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.633249 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.634480 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:24.635220 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:24.635296 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:24.635353 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:24.635750 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 02:18:24.635803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":111,"threads_started":1}
I20260808 02:18:24.636677 11787 tablet_service.cc:1467] Processing CreateTablet for tablet 101be4cb9297483cad383922563e96da (DEFAULT_TABLE table=test-table [id=e74be7688c33429c838ca8c1ea9ce4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:24.636759 11982 tablet_service.cc:1467] Processing CreateTablet for tablet 101be4cb9297483cad383922563e96da (DEFAULT_TABLE table=test-table [id=e74be7688c33429c838ca8c1ea9ce4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:24.636745 12165 tablet_service.cc:1467] Processing CreateTablet for tablet 101be4cb9297483cad383922563e96da (DEFAULT_TABLE table=test-table [id=e74be7688c33429c838ca8c1ea9ce4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:18:24.637043 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101be4cb9297483cad383922563e96da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:24.637106 11982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101be4cb9297483cad383922563e96da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:24.637106 12165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101be4cb9297483cad383922563e96da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:18:24.639744 12286 tablet_bootstrap.cc:492] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: Bootstrap starting.
I20260808 02:18:24.640410 12287 tablet_bootstrap.cc:492] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Bootstrap starting.
I20260808 02:18:24.640488 12286 tablet_bootstrap.cc:654] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:24.641063 12288 tablet_bootstrap.cc:492] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Bootstrap starting.
I20260808 02:18:24.641146 12287 tablet_bootstrap.cc:654] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:24.641234 12286 log.cc:824] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:24.641691 12287 log.cc:824] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:24.641723 12288 tablet_bootstrap.cc:654] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Neither blocks nor log segments found. Creating new log.
I20260808 02:18:24.642140 12286 tablet_bootstrap.cc:492] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: No bootstrap required, opened a new log
I20260808 02:18:24.642232 12286 ts_tablet_manager.cc:1397] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:18:24.642329 12288 log.cc:824] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:18:24.642521 12287 tablet_bootstrap.cc:492] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: No bootstrap required, opened a new log
I20260808 02:18:24.642622 12287 ts_tablet_manager.cc:1397] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:18:24.643160 12288 tablet_bootstrap.cc:492] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: No bootstrap required, opened a new log
I20260808 02:18:24.643281 12288 ts_tablet_manager.cc:1397] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:18:24.644265 12287 raft_consensus.cc:348] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:24.644399 12287 raft_consensus.cc:374] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:24.644423 12287 raft_consensus.cc:729] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a77976fe394189a86073d053138a4e, State: Initialized, Role: FOLLOWER
I20260808 02:18:24.644353 12286 raft_consensus.cc:348] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:24.644522 12286 raft_consensus.cc:374] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:24.644568 12286 raft_consensus.cc:729] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad3d3ac18bc4099bbc810ad172fe000, State: Initialized, Role: FOLLOWER
I20260808 02:18:24.644554 12287 consensus_queue.cc:260] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:24.644690 12286 consensus_queue.cc:260] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:24.644894 12287 ts_tablet_manager.cc:1428] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 02:18:24.644909 12077 heartbeater.cc:503] Master 127.28.177.190:43217 was elected leader, sending a full tablet report...
I20260808 02:18:24.645081 12286 ts_tablet_manager.cc:1428] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:18:24.645219 11881 heartbeater.cc:503] Master 127.28.177.190:43217 was elected leader, sending a full tablet report...
I20260808 02:18:24.645934 12288 raft_consensus.cc:348] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:24.646134 12288 raft_consensus.cc:374] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:18:24.646191 12288 raft_consensus.cc:729] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ce19437f51493c821e44eae8e3dba0, State: Initialized, Role: FOLLOWER
I20260808 02:18:24.646329 12288 consensus_queue.cc:260] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:24.646627 12288 ts_tablet_manager.cc:1428] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:18:24.646659 12262 heartbeater.cc:503] Master 127.28.177.190:43217 was elected leader, sending a full tablet report...
W20260808 02:18:24.682807 12078 tablet.cc:2369] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:18:24.768776 11882 tablet.cc:2369] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:18:24.847568 12263 tablet.cc:2369] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:18:25.002990 12294 raft_consensus.cc:482] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:18:25.003165 12294 raft_consensus.cc:504] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:25.003662 12294 leader_election.cc:288] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91a77976fe394189a86073d053138a4e (127.28.177.130:34945), 67ce19437f51493c821e44eae8e3dba0 (127.28.177.131:45101)
I20260808 02:18:25.007092 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:25.003991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:41069 (local address 127.28.177.130:34945)
0808 02:18:25.004079 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:25.004082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:25.004291 (+ 209us) server_negotiation.cc:407] Connection header received
0808 02:18:25.004322 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:25.004323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:25.004334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:25.004393 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:25.004611 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.005234 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.006281 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.006704 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.006810 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:25.006820 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:25.006968 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 02:18:25.007016 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:18:25.007124 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:25.004040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:51513 (local address 127.28.177.131:45101)
0808 02:18:25.004090 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:25.004092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:25.004198 (+ 106us) server_negotiation.cc:407] Connection header received
0808 02:18:25.004237 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:25.004238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:25.004250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:25.004317 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:18:25.004502 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.005104 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.006333 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.006706 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.006799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:25.006809 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:25.007021 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 02:18:25.007060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:18:25.007138 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:25.003889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:45101 (local address 127.28.177.129:51513)
0808 02:18:25.004099 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:25.004112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:25.004184 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:25.004319 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:25.004320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:25.004322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:25.004414 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:25.004418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.005224 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.005225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:25.006204 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:25.006206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.006822 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.006823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:25.006908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:25.006912 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:25.007005 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:18:25.007045 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":30,"thread_start_us":127,"threads_started":1}
I20260808 02:18:25.007177 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:25.003889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:34945 (local address 127.28.177.129:41069)
0808 02:18:25.004149 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:25.004158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:18:25.004249 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:18:25.004414 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:18:25.004415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:18:25.004416 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:18:25.004521 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:25.004523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.005326 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.005326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:25.006175 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:18:25.006178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.006810 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.006811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:18:25.006888 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:25.006892 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:18:25.006937 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 02:18:25.006999 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":20,"thread_start_us":95,"threads_started":1}
I20260808 02:18:25.007474 12197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101be4cb9297483cad383922563e96da" candidate_uuid: "cad3d3ac18bc4099bbc810ad172fe000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67ce19437f51493c821e44eae8e3dba0" is_pre_election: true
I20260808 02:18:25.007668 12197 raft_consensus.cc:2454] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad3d3ac18bc4099bbc810ad172fe000 in term 0.
I20260808 02:18:25.007718 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101be4cb9297483cad383922563e96da" candidate_uuid: "cad3d3ac18bc4099bbc810ad172fe000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a77976fe394189a86073d053138a4e" is_pre_election: true
I20260808 02:18:25.007858 12008 raft_consensus.cc:2454] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad3d3ac18bc4099bbc810ad172fe000 in term 0.
I20260808 02:18:25.008114 11715 leader_election.cc:302] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a77976fe394189a86073d053138a4e, cad3d3ac18bc4099bbc810ad172fe000; no voters:
I20260808 02:18:25.008267 12294 raft_consensus.cc:2781] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:18:25.008332 12294 raft_consensus.cc:482] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:18:25.008370 12294 raft_consensus.cc:3037] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:25.009281 12294 raft_consensus.cc:504] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:25.009449 12294 leader_election.cc:288] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [CANDIDATE]: Term 1 election: Requested vote from peers 91a77976fe394189a86073d053138a4e (127.28.177.130:34945), 67ce19437f51493c821e44eae8e3dba0 (127.28.177.131:45101)
I20260808 02:18:25.009622 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101be4cb9297483cad383922563e96da" candidate_uuid: "cad3d3ac18bc4099bbc810ad172fe000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a77976fe394189a86073d053138a4e"
I20260808 02:18:25.009654 12197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101be4cb9297483cad383922563e96da" candidate_uuid: "cad3d3ac18bc4099bbc810ad172fe000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67ce19437f51493c821e44eae8e3dba0"
I20260808 02:18:25.009723 12008 raft_consensus.cc:3037] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:25.009729 12197 raft_consensus.cc:3037] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:18:25.010504 12008 raft_consensus.cc:2454] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad3d3ac18bc4099bbc810ad172fe000 in term 1.
I20260808 02:18:25.010555 12197 raft_consensus.cc:2454] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad3d3ac18bc4099bbc810ad172fe000 in term 1.
I20260808 02:18:25.010751 11715 leader_election.cc:302] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a77976fe394189a86073d053138a4e, cad3d3ac18bc4099bbc810ad172fe000; no voters:
I20260808 02:18:25.010849 12294 raft_consensus.cc:2781] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:18:25.011013 12294 raft_consensus.cc:686] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [term 1 LEADER]: Becoming Leader. State: Replica: cad3d3ac18bc4099bbc810ad172fe000, State: Running, Role: LEADER
I20260808 02:18:25.011127 12294 consensus_queue.cc:237] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } }
I20260808 02:18:25.012272 11601 catalog_manager.cc:5495] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 reported cstate change: term changed from 0 to 1, leader changed from <none> to cad3d3ac18bc4099bbc810ad172fe000 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "cad3d3ac18bc4099bbc810ad172fe000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad3d3ac18bc4099bbc810ad172fe000" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 38043 } health_report { overall_health: HEALTHY } } }
I20260808 02:18:25.078708 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:25.075599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.28.177.129:38043)
0808 02:18:25.075680 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:18:25.075682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:18:25.075699 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:18:25.075750 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:18:25.075751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:18:25.075772 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:18:25.075859 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:18:25.076114 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.076632 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.077569 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:18:25.077786 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:18:25.077920 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:18:25.078257 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:18:25.078384 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:18:25.078505 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:18:25.078575 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:18:25.078602 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:18:25.080929 12197 raft_consensus.cc:1261] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 1 FOLLOWER]: Refusing update from remote peer cad3d3ac18bc4099bbc810ad172fe000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:25.080950 12008 raft_consensus.cc:1261] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 1 FOLLOWER]: Refusing update from remote peer cad3d3ac18bc4099bbc810ad172fe000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:18:25.081405 12294 consensus_queue.cc:1035] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67ce19437f51493c821e44eae8e3dba0" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:25.081627 12301 consensus_queue.cc:1035] T 101be4cb9297483cad383922563e96da P cad3d3ac18bc4099bbc810ad172fe000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a77976fe394189a86073d053138a4e" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:18:25.084025 12304 mvcc.cc:205] Tried to move back new op lower bound from 7316092948807921664 to 7316092948526309376. Current Snapshot: MvccSnapshot[applied={T|T < 7316092948807921664}]
I20260808 02:18:25.084352 12305 mvcc.cc:205] Tried to move back new op lower bound from 7316092948807921664 to 7316092948526309376. Current Snapshot: MvccSnapshot[applied={T|T < 7316092948807921664}]
I20260808 02:18:25.085348 12307 mvcc.cc:205] Tried to move back new op lower bound from 7316092948807921664 to 7316092948526309376. Current Snapshot: MvccSnapshot[applied={T|T < 7316092948807921664 or (T in {7316092948807921664})}]
I20260808 02:18:25.086967 11600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:25.087088 11600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:18:25.087970 11600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3f8eaef9d4141a8988fed4912457034: Sending DeleteTablet for 3 replicas of tablet 101be4cb9297483cad383922563e96da
I20260808 02:18:25.088351 12165 tablet_service.cc:1514] Processing DeleteTablet for tablet 101be4cb9297483cad383922563e96da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:25 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:47728
I20260808 02:18:25.088384 11787 tablet_service.cc:1514] Processing DeleteTablet for tablet 101be4cb9297483cad383922563e96da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:25 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:34212
I20260808 02:18:25.088413 11982 tablet_service.cc:1514] Processing DeleteTablet for tablet 101be4cb9297483cad383922563e96da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:18:25 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:39286
I20260808 02:18:25.088676 12317 tablet_replica.cc:330] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: stopping tablet replica
I20260808 02:18:25.088796 12317 raft_consensus.cc:2229] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:25.088901 12316 tablet_replica.cc:330] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: stopping tablet replica
I20260808 02:18:25.089037 12317 raft_consensus.cc:2258] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:25.089071 12316 raft_consensus.cc:2229] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:18:25.089097 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 11690
I20260808 02:18:25.105499 12316 raft_consensus.cc:2258] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:18:25.106673 12316 ts_tablet_manager.cc:1905] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:25.106935 12317 ts_tablet_manager.cc:1905] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:18:25.108979 12316 ts_tablet_manager.cc:1918] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:25.109103 12317 ts_tablet_manager.cc:1918] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:18:25.109191 12317 log.cc:1196] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/101be4cb9297483cad383922563e96da
I20260808 02:18:25.109200 12316 log.cc:1196] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/101be4cb9297483cad383922563e96da
I20260808 02:18:25.109495 12316 ts_tablet_manager.cc:1939] T 101be4cb9297483cad383922563e96da P 67ce19437f51493c821e44eae8e3dba0: Deleting consensus metadata
I20260808 02:18:25.109512 12317 ts_tablet_manager.cc:1939] T 101be4cb9297483cad383922563e96da P 91a77976fe394189a86073d053138a4e: Deleting consensus metadata
I20260808 02:18:25.110077 11578 catalog_manager.cc:4841] TS 67ce19437f51493c821e44eae8e3dba0 (127.28.177.131:45101): tablet 101be4cb9297483cad383922563e96da (table test-table [id=e74be7688c33429c838ca8c1ea9ce4d4]) successfully deleted
I20260808 02:18:25.110077 11576 catalog_manager.cc:4841] TS 91a77976fe394189a86073d053138a4e (127.28.177.130:34945): tablet 101be4cb9297483cad383922563e96da (table test-table [id=e74be7688c33429c838ca8c1ea9ce4d4]) successfully deleted
W20260808 02:18:25.111603 11578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:38043 (error 108)
I20260808 02:18:25.111874 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 11888
I20260808 02:18:25.112233 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:18:25.112110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:38043 (local address 127.0.0.1:34224)
0808 02:18:25.112164 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:18:25.112195 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:38043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 02:18:25.112346 11578 catalog_manager.cc:4584] TS cad3d3ac18bc4099bbc810ad172fe000 (127.28.177.129:38043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 101be4cb9297483cad383922563e96da: Network error: Client connection negotiation failed: client connection to 127.28.177.129:38043: connect: Connection refused (error 111)
I20260808 02:18:25.121343 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 12084
I20260808 02:18:25.131253 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 11551
2026-08-08T02:18:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2619 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:18:25 dist-test-slave-3h5h krb5kdc[12324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:18:25 dist-test-slave-3h5h krb5kdc[12324](info): set up 2 sockets
Aug 08 02:18:25 dist-test-slave-3h5h krb5kdc[12324](info): commencing operation
krb5kdc: starting...
W20260808 02:18:26.406710 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:18:26 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:18:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/rangeradmin_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/rangeradmin_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/rangerlookup_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/rangerlookup_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
I20260808 02:18:26.609576 29382 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfrqfAr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/postgres -l logfile start
2026-08-08 02:18:29.538 UTC [12369] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:18:29.538 UTC [12369] LOG: listening on IPv4 address "127.28.177.148", port 32841
2026-08-08 02:18:29.542 UTC [12369] LOG: listening on Unix socket "/tmp/.s.PGSQL.32841"
2026-08-08 02:18:29.565 UTC [12373] LOG: database system was shut down at 2026-08-08 02:18:27 UTC
2026-08-08 02:18:29.569 UTC [12369] LOG: database system is ready to accept connections
I20260808 02:18:30.785845 29382 mini_postgres.cc:96] Postgres bound to 32841
2026-08-08 02:18:30.793 UTC [12382] FATAL: database "slave" does not exist
127.28.177.148:32841 - accepting connections
I20260808 02:18:30.794471 29382 mini_ranger.cc:161] Starting Ranger...
I20260808 02:18:30.811903 29382 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:18:31.426672 29382 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:18:31.426779 29382 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-admin
2026-08-08 02:18:31,761 [I] DB FLAVOR :POSTGRES
2026-08-08 02:18:31,762 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:18:31,762 [I] Checking connection..
2026-08-08 02:18:31,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:18:32,061 [I] Checking connection passed.
2026-08-08 02:18:32,061 [I] --------- Verifying version history table ---------
2026-08-08 02:18:32,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:18:32,363 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:18:32,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:18:32,647 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:18:32,647 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:18:32,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:18:32.906 UTC [12469] WARNING: there is no transaction in progress
2026-08-08 02:18:32,916 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:18:32,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:18:33,169 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:18:33,169 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:18:33,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:18:33,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:18:33,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3h5h.c.gcp-upstream.internal','N') ;"
2026-08-08 02:18:33,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:18:34,266 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:18:34,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:18:34,542 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:18:34,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:18:34,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:18:35,102 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:18:35,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:18:36.279 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.298 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.319 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.338 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.357 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.378 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.449 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.457 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.475 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.487 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.505 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.522 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.533 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.541 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.550 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:36.559 UTC [12625] WARNING: there is no transaction in progress
2026-08-08 02:18:37,367 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:18:37,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:18:37,648 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:18:37,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:18:37,896 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:18:37,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:18:38,188 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:18:38,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:18:38,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:18:38,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3h5h.c.gcp-upstream.internal';"
2026-08-08 02:18:39,043 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:18:39,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:18:39,332 [I] DB_PATCHES have already been applied
I20260808 02:18:39.339046 29382 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfrqfAr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:18:39.339157 29382 mini_ranger.cc:193] Using host: 127.28.177.148
I20260808 02:18:39.343669 29382 mini_ranger.cc:227] Ranger admin URL: http://127.28.177.148:59347
Aug 08, 2026 2:18:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:18:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:18:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:18:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:18:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:18:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:18:40.697474 29382 mini_ranger.cc:160] Time spent starting Ranger: real 9.903s user 0.000s sys 0.013s
Aug 08, 2026 2:19:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:19:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 2:19:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:19:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:19:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:19:04.798048 29382 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/rangerkms_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/rangerkms_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.177.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.177.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:19:04 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155544, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:19:04.897061 29382 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:19:04.911607 29382 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:19:05.971678 29382 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:19:05.971760 29382 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms
I20260808 02:19:05.971774 29382 mini_ranger_kms.cc:227] Using postgres at 127.28.177.148:32841
2026-08-08 02:19:06,766 [I] DB FLAVOR :POSTGRES
2026-08-08 02:19:06,766 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:19:06,766 [I] Checking connection
2026-08-08 02:19:06,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:19:07,149 [I] connection success
2026-08-08 02:19:07,149 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:19:07,149 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:19:07,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:19:07,436 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:19:07,436 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:19:07,436 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:19:07,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:32841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:19:07,737 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:19:08.271890 29382 mini_ranger_kms.cc:310] Created kms service
I20260808 02:19:08.500196 29382 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:19:08.550411 29382 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:19:08 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155544, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:19:09.024540 29382 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:19:09.024937 29382 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:19:09.024983 29382 mini_ranger_kms.cc:240] Using host: 127.28.177.148
I20260808 02:19:09.029757 29382 mini_ranger_kms.cc:276] Ranger KMS PID: 12888
I20260808 02:19:09.029906 29382 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.177.148:40087
Aug 08, 2026 2:19:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:19:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:19:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:19:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:19:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:19:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 02:19:12.076248 29382 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.179s user 0.000s sys 0.765s
I20260808 02:19:12.076386 29382 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:19:12.076503 29382 mini_ranger_kms.cc:401] 127.28.177.148:40087/kms/v1/keys
Aug 08 02:19:12 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155552, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.177.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 02:19:12 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155552, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.177.148@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
Aug 08 02:19:13 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155552, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.177.148@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
Aug 08 02:19:15 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:19:15.774513 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:40953
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:40953
--ranger_config_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:40087/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:19:15.910262 12946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:19:15.910513 12946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:19:15.910539 12946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:19:15.911782 12946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:19:15.911866 12946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:19:15.911899 12946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:19:15.911962 12946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:19:15.911998 12946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:19:15.913982 12946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.177.148:40087/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:40953
--ranger_config_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:19:15.914290 12946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:19:15.914556 12946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:19:15.921092 12955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:19:15.921202 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:15.921250 12946 server_base.cc:1034] running on GCE node
W20260808 02:19:15.921581 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:15.921916 12946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:19:15.922289 12946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:19:15.923498 12946 hybrid_clock.cc:648] HybridClock initialized: now 1786155555923471 us; error 43 us; skew 500 ppm
Aug 08 02:19:15 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155555, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:19:15.927146 12946 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:19:15.927830 12946 webserver.cc:460] Webserver started at http://127.28.177.190:43049/ using document root <none> and password file <none>
I20260808 02:19:15.928176 12946 fs_manager.cc:359] Metadata directory not provided
I20260808 02:19:15.928244 12946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:19:15.928413 12946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:19:15 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155555, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:19:15.987629 12946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "7d8a2717ded2414bbfe4c88ecab194e4"
format_stamp: "Formatted at 2026-08-08 02:19:15 on dist-test-slave-3h5h"
server_key: "7ff6dc25761c7b86eb64f345bea6c20f"
server_key_iv: "4b32a716748f17ac46c73eaaa5a07c6a"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:15.988188 12946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "7d8a2717ded2414bbfe4c88ecab194e4"
format_stamp: "Formatted at 2026-08-08 02:19:15 on dist-test-slave-3h5h"
server_key: "7ff6dc25761c7b86eb64f345bea6c20f"
server_key_iv: "4b32a716748f17ac46c73eaaa5a07c6a"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:15.990602 12946 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:19:16.007879 12975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:16.008132 12946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:19:16.008244 12946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "7d8a2717ded2414bbfe4c88ecab194e4"
format_stamp: "Formatted at 2026-08-08 02:19:15 on dist-test-slave-3h5h"
server_key: "7ff6dc25761c7b86eb64f345bea6c20f"
server_key_iv: "4b32a716748f17ac46c73eaaa5a07c6a"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:16.008395 12946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:19:16.028194 12946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:19:16.030787 12946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:19:16.031068 12946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:19:16.044724 12946 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:40953
I20260808 02:19:16.044889 13054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:40953 every 8 connection(s)
I20260808 02:19:16.045344 12946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:19:16.046548 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:19:16.049768 13056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4: Bootstrap starting.
I20260808 02:19:16.050702 13056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4: Neither blocks nor log segments found. Creating new log.
I20260808 02:19:16.051342 13056 log.cc:824] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:19:16.052276 13056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4: No bootstrap required, opened a new log
I20260808 02:19:16.054395 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 12946
I20260808 02:19:16.054527 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:19:16.054764 29382 external_mini_cluster.cc:1428] Setting key 55dcf60f5c3651acc14ed96f948ce825
I20260808 02:19:16.055964 13056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 40953 } }
I20260808 02:19:16.056162 13056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:19:16.056229 13056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8a2717ded2414bbfe4c88ecab194e4, State: Initialized, Role: FOLLOWER
I20260808 02:19:16.056427 13056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 40953 } }
I20260808 02:19:16.056536 13056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:19:16.056571 13056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:19:16.056619 13056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:19:16.057716 13056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 40953 } }
I20260808 02:19:16.057847 13056 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8a2717ded2414bbfe4c88ecab194e4; no voters:
I20260808 02:19:16.058058 13056 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:19:16.058154 13061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:19:16.058379 13061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7d8a2717ded2414bbfe4c88ecab194e4, State: Running, Role: LEADER
I20260808 02:19:16.058516 13056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:19:16.058533 13061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 40953 } }
I20260808 02:19:16.058990 13061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 40953 } } }
I20260808 02:19:16.059108 13061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:19:16.059340 13061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8a2717ded2414bbfe4c88ecab194e4. Latest consensus state: current_term: 1 leader_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8a2717ded2414bbfe4c88ecab194e4" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 40953 } } }
I20260808 02:19:16.059432 13061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:19:16.060415 13070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:19:16.062491 13056 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:19:16.061944 13070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 02:19:16 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:19:16.075366 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:16.055750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.28.177.190:40953)
0808 02:19:16.056091 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:16.056104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:16.056179 (+ 75us) server_negotiation.cc:407] Connection header received
0808 02:19:16.056319 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:16.056327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:16.056397 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:16.056569 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:19:16.057508 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:16.059752 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:16.060907 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:16.061756 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:16.061886 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:16.067313 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:19:16.067490 (+ 177us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:19:16.067490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:19:16.067817 (+ 327us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:19:16.072081 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:16.072635 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:16.072638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:16.072639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:16.072911 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:16.073223 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:16.073224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:16.073225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:16.073621 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:19:16.074428 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:16.074510 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:19:16.074681 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":122,"threads_started":1}
I20260808 02:19:16.079213 13070 catalog_manager.cc:1347] Generated new cluster ID: be5a27419657415ea79b6eec8509d73d
I20260808 02:19:16.079299 13070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:19:16.089793 13070 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:19:16.091003 13070 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:19:16.098628 13070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4: Generated new TSK 0
I20260808 02:19:16.099035 13070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:19:17.108445 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:40087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:40953
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:19:17.237895 13109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:19:17.238145 13109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:19:17.238174 13109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:19:17.239612 13109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:19:17.239665 13109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:19:17.239697 13109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:19:17.241559 13109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.177.148:40087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:40953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:19:17.241863 13109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:19:17.242192 13109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:19:17.249843 13118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:19:17.249930 13117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:17.250322 13109 server_base.cc:1034] running on GCE node
W20260808 02:19:17.250463 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:17.250753 13109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:19:17.251124 13109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:19:17.252303 13109 hybrid_clock.cc:648] HybridClock initialized: now 1786155557252284 us; error 37 us; skew 500 ppm
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:19:17.256237 13109 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:19:17.256939 13109 webserver.cc:460] Webserver started at http://127.28.177.129:44355/ using document root <none> and password file <none>
I20260808 02:19:17.257309 13109 fs_manager.cc:359] Metadata directory not provided
I20260808 02:19:17.257388 13109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:19:17.257561 13109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:19:17.274781 13109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "61d115085d05464ba32db496e516f22b"
format_stamp: "Formatted at 2026-08-08 02:19:17 on dist-test-slave-3h5h"
server_key: "c8c53ab6e700d6e7f6428b2ccb83bd25"
server_key_iv: "ea6b10aceab137e5a6ce23a684d8608b"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:17.275449 13109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "61d115085d05464ba32db496e516f22b"
format_stamp: "Formatted at 2026-08-08 02:19:17 on dist-test-slave-3h5h"
server_key: "c8c53ab6e700d6e7f6428b2ccb83bd25"
server_key_iv: "ea6b10aceab137e5a6ce23a684d8608b"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:17.277724 13109 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260808 02:19:17.296772 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:17.301272 13109 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260808 02:19:17.301829 13109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "61d115085d05464ba32db496e516f22b"
format_stamp: "Formatted at 2026-08-08 02:19:17 on dist-test-slave-3h5h"
server_key: "c8c53ab6e700d6e7f6428b2ccb83bd25"
server_key_iv: "ea6b10aceab137e5a6ce23a684d8608b"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:17.302367 13109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:19:17.327553 13109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:19:17.330540 13109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:19:17.330823 13109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:19:17.331254 13109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:19:17.331688 13109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:19:17.331732 13109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:17.331763 13109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:19:17.331780 13109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:17.351701 13109 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:43841
I20260808 02:19:17.352295 13109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:19:17.354403 13284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:43841 every 8 connection(s)
I20260808 02:19:17.357367 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 13109
I20260808 02:19:17.357494 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:19:17.357715 29382 external_mini_cluster.cc:1428] Setting key e2ef109ccd2afccddc68a106e1a9970f
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:19:17.371833 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:17.354866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:44883 (local address 127.28.177.190:40953)
0808 02:19:17.356378 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:17.356380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:17.356387 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:19:17.356412 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:17.356413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:17.356423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:17.356515 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:19:17.360998 (+ 4483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.361841 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.363732 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.364086 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.364173 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:17.367679 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:19:17.367682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:19:17.367682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:19:17.367693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:19:17.369862 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:17.370546 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:17.370548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:17.370549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:17.370656 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:17.370958 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:17.370959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:17.370960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:17.371119 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:19:17.371205 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:17.371623 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 02:19:17.371728 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":92,"threads_started":1}
I20260808 02:19:17.372046 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:17.354395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:40953 (local address 127.28.177.129:44883)
0808 02:19:17.355116 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:17.355174 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:17.356205 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:17.356603 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:17.356607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:17.357095 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:19:17.358534 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:17.358544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.362026 (+ 3482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.362027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:17.363613 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:17.363618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.364258 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.364259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:17.364495 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:17.365307 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:19:17.365319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:19:17.367516 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:19:17.369976 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:19:17.369986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:19:17.369990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:19:17.370430 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:19:17.370731 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:19:17.370732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:19:17.370733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:19:17.370880 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:19:17.371236 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:19:17.371243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:19:17.371437 (+ 194us) client_negotiation.cc:769] Sending connection context
0808 02:19:17.371584 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 02:19:17.371755 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":189,"threads_started":1}
I20260808 02:19:17.373100 13285 heartbeater.cc:347] Connected to a master server at 127.28.177.190:40953
I20260808 02:19:17.373221 13285 heartbeater.cc:464] Registering TS with master...
I20260808 02:19:17.373519 13285 heartbeater.cc:511] Master 127.28.177.190:40953 requested a full tablet report, sending...
I20260808 02:19:17.374423 12997 ts_manager.cc:194] Registered new tserver with Master: 61d115085d05464ba32db496e516f22b (127.28.177.129:43841)
I20260808 02:19:17.375494 12997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:44883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:19:17.400579 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:40087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:40953
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:19:17.632515 13292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:19:17.632953 13292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:19:17.633018 13292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:19:17.635599 13292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:19:17.635684 13292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:19:17.635741 13292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:19:17.638793 13292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.177.148:40087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:40953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:19:17.639138 13292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:19:17.639503 13292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:19:17.650722 13308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:17.651019 13292 server_base.cc:1034] running on GCE node
W20260808 02:19:17.651610 13305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:19:17.652118 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:17.652405 13292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:19:17.652805 13292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:19:17.654016 13292 hybrid_clock.cc:648] HybridClock initialized: now 1786155557653959 us; error 70 us; skew 500 ppm
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:19:17.659113 13292 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:19:17.659927 13292 webserver.cc:460] Webserver started at http://127.28.177.130:41959/ using document root <none> and password file <none>
I20260808 02:19:17.660399 13292 fs_manager.cc:359] Metadata directory not provided
I20260808 02:19:17.660595 13292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:19:17.660882 13292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:19:17.683468 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "ed27b94f95c34bb8a76a7fac99ce64e8"
format_stamp: "Formatted at 2026-08-08 02:19:17 on dist-test-slave-3h5h"
server_key: "4c48d4751491ebe097d3c233e7b6a3d0"
server_key_iv: "7e49df07ee65b9833411e360fea6363d"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:17.684027 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "ed27b94f95c34bb8a76a7fac99ce64e8"
format_stamp: "Formatted at 2026-08-08 02:19:17 on dist-test-slave-3h5h"
server_key: "4c48d4751491ebe097d3c233e7b6a3d0"
server_key_iv: "7e49df07ee65b9833411e360fea6363d"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:17.690397 13292 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.002s
I20260808 02:19:17.709071 13317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:17.709455 13292 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260808 02:19:17.709606 13292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "ed27b94f95c34bb8a76a7fac99ce64e8"
format_stamp: "Formatted at 2026-08-08 02:19:17 on dist-test-slave-3h5h"
server_key: "4c48d4751491ebe097d3c233e7b6a3d0"
server_key_iv: "7e49df07ee65b9833411e360fea6363d"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:17.709718 13292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:19:17.737968 13292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:19:17.741117 13292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:19:17.741294 13292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:19:17.741552 13292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:19:17.741952 13292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:19:17.741989 13292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:17.742020 13292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:19:17.742038 13292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:17.761348 13292 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:37983
I20260808 02:19:17.761934 13292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:19:17.766520 13478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:37983 every 8 connection(s)
I20260808 02:19:17.770717 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 13292
I20260808 02:19:17.770838 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:19:17.771035 29382 external_mini_cluster.cc:1428] Setting key 6662fe5f3ebbc1cabdf9e819cd9c89fa
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
I20260808 02:19:17.789407 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:17.767314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:56049 (local address 127.28.177.190:40953)
0808 02:19:17.767371 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:17.767373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:17.767876 (+ 503us) server_negotiation.cc:407] Connection header received
0808 02:19:17.769092 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:17.769094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:17.769113 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:17.769180 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:19:17.771216 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.772028 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.775049 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.775521 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.775639 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:17.781128 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:19:17.781138 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:19:17.781138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:19:17.781149 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:19:17.783496 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:17.784431 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:17.784432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:17.784433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:17.784549 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:17.784975 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:17.784977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:17.784977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:17.785146 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:19:17.785205 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:17.789176 (+ 3971us) server_negotiation.cc:299] Negotiation successful
0808 02:19:17.789297 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:19:17.790421 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:17.767260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:40953 (local address 127.28.177.130:56049)
0808 02:19:17.767657 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:17.767716 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:17.768942 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:17.769269 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:17.769274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:17.769653 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:19:17.771085 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:17.771091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.772450 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.772451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:17.774921 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:17.774926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:17.777215 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:17.777217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:17.777477 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:17.778499 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:19:17.778512 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:19:17.780932 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:19:17.783685 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:19:17.783705 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:19:17.783708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:19:17.784274 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:19:17.784654 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:19:17.784655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:19:17.784655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:19:17.784873 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:19:17.788699 (+ 3826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:19:17.788710 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:19:17.788961 (+ 251us) client_negotiation.cc:769] Sending connection context
0808 02:19:17.789849 (+ 888us) client_negotiation.cc:240] Negotiation successful
0808 02:19:17.790078 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260808 02:19:17.791493 13479 heartbeater.cc:347] Connected to a master server at 127.28.177.190:40953
I20260808 02:19:17.791612 13479 heartbeater.cc:464] Registering TS with master...
I20260808 02:19:17.791836 13479 heartbeater.cc:511] Master 127.28.177.190:40953 requested a full tablet report, sending...
I20260808 02:19:17.792349 12997 ts_manager.cc:194] Registered new tserver with Master: ed27b94f95c34bb8a76a7fac99ce64e8 (127.28.177.130:37983)
I20260808 02:19:17.792886 12997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:56049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:19:17.824821 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:40087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:40953
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:19:17.980130 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:19:17.980512 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:19:17.980571 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:19:17.982601 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:19:17.982708 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:19:17.982766 13490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:19:17.984901 13490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.177.148:40087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:40953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:19:17.985240 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:19:17.985532 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:19:17.992467 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:19:17.992781 13499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:19:17.993111 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:19:17.993450 13490 server_base.cc:1034] running on GCE node
I20260808 02:19:17.993714 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:19:17.994035 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:19:17.995255 13490 hybrid_clock.cc:648] HybridClock initialized: now 1786155557995226 us; error 43 us; skew 500 ppm
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:19:17.999663 13490 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:19:18.000378 13490 webserver.cc:460] Webserver started at http://127.28.177.131:39385/ using document root <none> and password file <none>
I20260808 02:19:18.000669 13490 fs_manager.cc:359] Metadata directory not provided
I20260808 02:19:18.000756 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:19:18.000967 13490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:19:17 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:19:18.015448 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "321c5620e2f94cc4a71d9dcc341ab645"
format_stamp: "Formatted at 2026-08-08 02:19:18 on dist-test-slave-3h5h"
server_key: "4559af1b71cd1a2a3da3b3f26dfd6643"
server_key_iv: "1c8d372243dd554436714185f3c9776e"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:18.016114 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "321c5620e2f94cc4a71d9dcc341ab645"
format_stamp: "Formatted at 2026-08-08 02:19:18 on dist-test-slave-3h5h"
server_key: "4559af1b71cd1a2a3da3b3f26dfd6643"
server_key_iv: "1c8d372243dd554436714185f3c9776e"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:18.023631 13490 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.002s
I20260808 02:19:18.041132 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:18.041373 13490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:19:18.041483 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "321c5620e2f94cc4a71d9dcc341ab645"
format_stamp: "Formatted at 2026-08-08 02:19:18 on dist-test-slave-3h5h"
server_key: "4559af1b71cd1a2a3da3b3f26dfd6643"
server_key_iv: "1c8d372243dd554436714185f3c9776e"
server_key_version: "kuduclusterkey@0"
I20260808 02:19:18.041594 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:19:18.064288 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:19:18.067459 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:19:18.067644 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:19:18.067934 13490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:19:18.068468 13490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:19:18.068527 13490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:18.068619 13490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:19:18.068651 13490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:19:18.091663 13490 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:34677
I20260808 02:19:18.092224 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:19:18.093515 13664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:34677 every 8 connection(s)
I20260808 02:19:18.094393 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 13490
I20260808 02:19:18.094542 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:19:18.094883 29382 external_mini_cluster.cc:1428] Setting key 6f7385315be73000178999d847d74c69
Aug 08 02:19:18 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:19:18.110666 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.097522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:47917 (local address 127.28.177.190:40953)
0808 02:19:18.097598 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.097600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.097826 (+ 226us) server_negotiation.cc:407] Connection header received
0808 02:19:18.099051 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.099053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.099066 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.099156 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:19:18.100529 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.101652 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.103141 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.103568 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.103673 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.106822 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:19:18.106823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:19:18.106824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:19:18.106845 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:19:18.108870 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:18.109524 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:18.109526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:18.109526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:18.109600 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:18.109903 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:18.109904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:18.109905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:18.110064 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:19:18.110149 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.110461 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.110557 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:19:18.110946 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.097078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:40953 (local address 127.28.177.131:47917)
0808 02:19:18.097532 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.097594 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.098880 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.099274 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:18.099280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:18.099839 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:19:18.100379 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.100387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.101803 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.101804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.103018 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.103021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.103699 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.103700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.103876 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.104779 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:19:18.104789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:19:18.106717 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:19:18.109134 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:19:18.109142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:19:18.109145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:19:18.109409 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:19:18.109686 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:19:18.109687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:19:18.109687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:19:18.109823 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:19:18.110169 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:19:18.110175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:19:18.110313 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 02:19:18.110421 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 02:19:18.110593 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":230,"threads_started":1}
I20260808 02:19:18.111697 13665 heartbeater.cc:347] Connected to a master server at 127.28.177.190:40953
I20260808 02:19:18.111821 13665 heartbeater.cc:464] Registering TS with master...
I20260808 02:19:18.112022 13665 heartbeater.cc:511] Master 127.28.177.190:40953 requested a full tablet report, sending...
I20260808 02:19:18.112525 12997 ts_manager.cc:194] Registered new tserver with Master: 321c5620e2f94cc4a71d9dcc341ab645 (127.28.177.131:34677)
I20260808 02:19:18.112977 12997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:47917
I20260808 02:19:18.118113 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:19:18.128118 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.119774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.28.177.190:40953)
0808 02:19:18.119852 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.119854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.119969 (+ 115us) server_negotiation.cc:407] Connection header received
0808 02:19:18.120039 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.120040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.120053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.120123 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:19:18.120859 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.121753 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.122682 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.122972 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.123067 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.124521 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:19:18.124522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:19:18.124522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:19:18.124531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:19:18.126651 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:18.127010 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:18.127011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:18.127011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:18.127121 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:19:18.127353 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:19:18.127354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:19:18.127354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:19:18.127532 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:19:18.127670 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.127838 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.127955 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:19:18.130373 12997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:19:18.133590 12997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:19:18.144819 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.138357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.28.177.129:43841)
0808 02:19:18.139166 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.139171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.139592 (+ 421us) server_negotiation.cc:407] Connection header received
0808 02:19:18.139685 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.139687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.139706 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.139822 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:19:18.140034 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.141958 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.143673 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.144318 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.144416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.144484 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.144644 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.144687 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":73,"threads_started":1}
I20260808 02:19:18.144770 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.137739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:43841 (local address 127.0.0.1:40600)
0808 02:19:18.139518 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.139531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.139598 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.139899 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:18.139900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:18.139902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:19:18.139935 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.139936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.142071 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.142072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.143580 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.143582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.144414 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.144415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.144506 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.144521 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:19:18.144624 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:19:18.144687 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"mutex_wait_us":51,"thread_start_us":193,"threads_started":1}
I20260808 02:19:18.145123 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.137723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:34677 (local address 127.0.0.1:51910)
0808 02:19:18.139176 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.139227 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.139356 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.139815 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:18.139818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:18.139821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:19:18.139909 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.139911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.141956 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.141957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.143344 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.143347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.144641 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.144642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.144765 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.144784 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 02:19:18.144942 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 02:19:18.145014 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"mutex_wait_us":2,"thread_start_us":312,"threads_started":1}
I20260808 02:19:18.145466 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.138543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.28.177.131:34677)
0808 02:19:18.139357 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.139362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.139377 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:19:18.139634 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.139639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.139678 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.139808 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:19:18.140942 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.141855 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.143494 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.144381 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.144516 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.144579 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.145263 (+ 684us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.145324 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":183,"threads_started":1}
I20260808 02:19:18.146243 13200 tablet_service.cc:1467] Processing CreateTablet for tablet 728f1cae91b34aafb72cc2f895cf9864 (DEFAULT_TABLE table=test-table [id=70b5f2e74a7849868a4f5114916f6877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:19:18.146690 13200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728f1cae91b34aafb72cc2f895cf9864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:19:18.149816 13579 tablet_service.cc:1467] Processing CreateTablet for tablet 728f1cae91b34aafb72cc2f895cf9864 (DEFAULT_TABLE table=test-table [id=70b5f2e74a7849868a4f5114916f6877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:19:18.150323 13579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728f1cae91b34aafb72cc2f895cf9864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:19:18.150930 13690 tablet_bootstrap.cc:492] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: Bootstrap starting.
I20260808 02:19:18.152518 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.137723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37983 (local address 127.0.0.1:60856)
0808 02:19:18.144967 (+ 7244us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.144980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.145041 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.145509 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:18.145510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:18.145513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:19:18.145562 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.145564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.147092 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.147094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.149482 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.149486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.151995 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.151996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.152174 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.152190 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 02:19:18.152278 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:19:18.152367 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7199,"mutex_wait_us":1058,"thread_start_us":252,"threads_started":1}
I20260808 02:19:18.152745 13690 tablet_bootstrap.cc:654] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: Neither blocks nor log segments found. Creating new log.
I20260808 02:19:18.152944 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.137996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.28.177.130:37983)
0808 02:19:18.138420 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.138426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.145115 (+ 6689us) server_negotiation.cc:407] Connection header received
0808 02:19:18.145219 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.145223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.145244 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.145333 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:19:18.145653 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.146969 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.149794 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.151521 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.151823 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.151923 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.152703 (+ 780us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.152759 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
I20260808 02:19:18.153523 13690 log.cc:824] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:19:18.154085 13364 tablet_service.cc:1467] Processing CreateTablet for tablet 728f1cae91b34aafb72cc2f895cf9864 (DEFAULT_TABLE table=test-table [id=70b5f2e74a7849868a4f5114916f6877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:19:18.154626 13690 tablet_bootstrap.cc:492] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: No bootstrap required, opened a new log
I20260808 02:19:18.154724 13690 ts_tablet_manager.cc:1397] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260808 02:19:18.154747 13364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728f1cae91b34aafb72cc2f895cf9864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:19:18.159252 13690 raft_consensus.cc:348] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.159482 13690 raft_consensus.cc:374] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:19:18.159543 13690 raft_consensus.cc:729] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d115085d05464ba32db496e516f22b, State: Initialized, Role: FOLLOWER
I20260808 02:19:18.159708 13690 consensus_queue.cc:260] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.160205 13690 ts_tablet_manager.cc:1428] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260808 02:19:18.160432 13285 heartbeater.cc:503] Master 127.28.177.190:40953 was elected leader, sending a full tablet report...
I20260808 02:19:18.160612 13692 tablet_bootstrap.cc:492] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Bootstrap starting.
I20260808 02:19:18.162245 13696 tablet_bootstrap.cc:492] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Bootstrap starting.
I20260808 02:19:18.162899 13692 tablet_bootstrap.cc:654] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Neither blocks nor log segments found. Creating new log.
I20260808 02:19:18.163550 13692 log.cc:824] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Log is configured to *not* fsync() on all Append() calls
I20260808 02:19:18.164616 13696 tablet_bootstrap.cc:654] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Neither blocks nor log segments found. Creating new log.
I20260808 02:19:18.164664 13692 tablet_bootstrap.cc:492] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: No bootstrap required, opened a new log
I20260808 02:19:18.164755 13692 ts_tablet_manager.cc:1397] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260808 02:19:18.165841 13696 log.cc:824] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:19:18.167524 13696 tablet_bootstrap.cc:492] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: No bootstrap required, opened a new log
I20260808 02:19:18.167656 13696 ts_tablet_manager.cc:1397] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 02:19:18.168677 13692 raft_consensus.cc:348] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.168923 13692 raft_consensus.cc:374] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:19:18.168983 13692 raft_consensus.cc:729] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321c5620e2f94cc4a71d9dcc341ab645, State: Initialized, Role: FOLLOWER
I20260808 02:19:18.169152 13692 consensus_queue.cc:260] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.169627 13692 ts_tablet_manager.cc:1428] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260808 02:19:18.169780 13665 heartbeater.cc:503] Master 127.28.177.190:40953 was elected leader, sending a full tablet report...
I20260808 02:19:18.173126 13696 raft_consensus.cc:348] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.173506 13696 raft_consensus.cc:374] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:19:18.173609 13696 raft_consensus.cc:729] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27b94f95c34bb8a76a7fac99ce64e8, State: Initialized, Role: FOLLOWER
I20260808 02:19:18.173867 13696 consensus_queue.cc:260] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.174428 13696 ts_tablet_manager.cc:1428] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 02:19:18.174584 13479 heartbeater.cc:503] Master 127.28.177.190:40953 was elected leader, sending a full tablet report...
W20260808 02:19:18.266855 13481 tablet.cc:2369] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:19:18.314354 13701 raft_consensus.cc:482] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:19:18.314524 13701 raft_consensus.cc:504] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.314955 13701 leader_election.cc:288] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 321c5620e2f94cc4a71d9dcc341ab645 (127.28.177.131:34677), 61d115085d05464ba32db496e516f22b (127.28.177.129:43841)
I20260808 02:19:18.321667 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.315967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:34699 (local address 127.28.177.131:34677)
0808 02:19:18.316016 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.316019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.316230 (+ 211us) server_negotiation.cc:407] Connection header received
0808 02:19:18.316347 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.316348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.316360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.316392 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:19:18.316810 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.317691 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.319807 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.320684 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.320780 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.320792 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.321529 (+ 737us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.321583 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:19:18.322039 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.315431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:47221 (local address 127.28.177.129:43841)
0808 02:19:18.315483 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.315485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.315835 (+ 350us) server_negotiation.cc:407] Connection header received
0808 02:19:18.315951 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.315953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.315965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.316008 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:19:18.316353 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.317131 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.319042 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.321436 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.321506 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.321515 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.321916 (+ 401us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.321967 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:19:18.322465 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.315538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:34677 (local address 127.28.177.130:34699)
0808 02:19:18.316149 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.316161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.316263 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.316493 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:18.316494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:18.316496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:19:18.316688 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.316690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.317930 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.317931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.319623 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.319626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.321221 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.321222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.321391 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.321398 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:19:18.322037 (+ 639us) client_negotiation.cc:240] Negotiation successful
0808 02:19:18.322237 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":14,"thread_start_us":290,"threads_started":1}
I20260808 02:19:18.322530 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.315270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:43841 (local address 127.28.177.130:47221)
0808 02:19:18.315553 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.315566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.315871 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.316079 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:19:18.316080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:19:18.316082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:19:18.316262 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.316265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.317233 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.317234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.318627 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:19:18.318631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.321590 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.321591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:19:18.321806 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.321811 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:19:18.322162 (+ 351us) client_negotiation.cc:240] Negotiation successful
0808 02:19:18.322212 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
I20260808 02:19:18.323107 13597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f1cae91b34aafb72cc2f895cf9864" candidate_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321c5620e2f94cc4a71d9dcc341ab645" is_pre_election: true
I20260808 02:19:18.323220 13229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f1cae91b34aafb72cc2f895cf9864" candidate_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d115085d05464ba32db496e516f22b" is_pre_election: true
I20260808 02:19:18.323334 13597 raft_consensus.cc:2454] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed27b94f95c34bb8a76a7fac99ce64e8 in term 0.
I20260808 02:19:18.323350 13229 raft_consensus.cc:2454] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed27b94f95c34bb8a76a7fac99ce64e8 in term 0.
I20260808 02:19:18.323734 13327 leader_election.cc:302] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d115085d05464ba32db496e516f22b, ed27b94f95c34bb8a76a7fac99ce64e8; no voters:
I20260808 02:19:18.323904 13701 raft_consensus.cc:2781] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:19:18.323956 13701 raft_consensus.cc:482] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:19:18.323977 13701 raft_consensus.cc:3037] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:19:18.325886 13701 raft_consensus.cc:504] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.326043 13701 leader_election.cc:288] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [CANDIDATE]: Term 1 election: Requested vote from peers 321c5620e2f94cc4a71d9dcc341ab645 (127.28.177.131:34677), 61d115085d05464ba32db496e516f22b (127.28.177.129:43841)
I20260808 02:19:18.326333 13229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f1cae91b34aafb72cc2f895cf9864" candidate_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d115085d05464ba32db496e516f22b"
I20260808 02:19:18.326418 13229 raft_consensus.cc:3037] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:19:18.327878 13229 raft_consensus.cc:2454] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed27b94f95c34bb8a76a7fac99ce64e8 in term 1.
I20260808 02:19:18.328081 13327 leader_election.cc:302] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d115085d05464ba32db496e516f22b, ed27b94f95c34bb8a76a7fac99ce64e8; no voters:
I20260808 02:19:18.328200 13597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f1cae91b34aafb72cc2f895cf9864" candidate_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321c5620e2f94cc4a71d9dcc341ab645"
I20260808 02:19:18.328295 13597 raft_consensus.cc:3037] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:19:18.328920 13701 raft_consensus.cc:2781] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:19:18.329165 13701 raft_consensus.cc:686] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 1 LEADER]: Becoming Leader. State: Replica: ed27b94f95c34bb8a76a7fac99ce64e8, State: Running, Role: LEADER
I20260808 02:19:18.329317 13701 consensus_queue.cc:237] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } }
I20260808 02:19:18.329852 13597 raft_consensus.cc:2454] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed27b94f95c34bb8a76a7fac99ce64e8 in term 1.
I20260808 02:19:18.330546 12997 catalog_manager.cc:5495] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed27b94f95c34bb8a76a7fac99ce64e8 (127.28.177.130). New cstate: current_term: 1 leader_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed27b94f95c34bb8a76a7fac99ce64e8" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 } health_report { overall_health: UNKNOWN } } }
W20260808 02:19:18.344310 13666 tablet.cc:2369] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:19:18.354398 13287 tablet.cc:2369] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:19:18 dist-test-slave-3h5h krb5kdc[12324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:19:18.387753 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.382830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.28.177.130:37983)
0808 02:19:18.382877 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:19:18.382879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:19:18.384051 (+ 1172us) server_negotiation.cc:407] Connection header received
0808 02:19:18.384107 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:19:18.384108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:19:18.384140 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:19:18.384214 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:19:18.384464 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.385150 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.386309 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:19:18.386558 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:19:18.386655 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:19:18.386803 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:19:18.386948 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:19:18.387093 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:19:18.387364 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:19:18.387389 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:19:18.390426 13229 raft_consensus.cc:1261] T 728f1cae91b34aafb72cc2f895cf9864 P 61d115085d05464ba32db496e516f22b [term 1 FOLLOWER]: Refusing update from remote peer ed27b94f95c34bb8a76a7fac99ce64e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:19:18.390450 13597 raft_consensus.cc:1261] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 1 FOLLOWER]: Refusing update from remote peer ed27b94f95c34bb8a76a7fac99ce64e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:19:18.390942 13704 consensus_queue.cc:1035] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321c5620e2f94cc4a71d9dcc341ab645" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:19:18.391080 13701 consensus_queue.cc:1035] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d115085d05464ba32db496e516f22b" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 43841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:19:18.396986 13707 mvcc.cc:205] Tried to move back new op lower bound from 7316093167163146240 to 7316093166917947392. Current Snapshot: MvccSnapshot[applied={T|T < 7316093167163146240}]
I20260808 02:19:18.406697 13002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:19:18.406826 13002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:19:18.407796 13002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d8a2717ded2414bbfe4c88ecab194e4: Sending DeleteTablet for 3 replicas of tablet 728f1cae91b34aafb72cc2f895cf9864
I20260808 02:19:18.408319 13200 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f1cae91b34aafb72cc2f895cf9864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:19:18 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:40600
I20260808 02:19:18.409118 13364 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f1cae91b34aafb72cc2f895cf9864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:19:18 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:60856
I20260808 02:19:18.409759 13579 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f1cae91b34aafb72cc2f895cf9864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:19:18 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:51910
I20260808 02:19:18.409885 13708 mvcc.cc:205] Tried to move back new op lower bound from 7316093167163146240 to 7316093166917947392. Current Snapshot: MvccSnapshot[applied={T|T < 7316093167163146240}]
I20260808 02:19:18.410418 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 13109
I20260808 02:19:18.422529 13720 tablet_replica.cc:330] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: stopping tablet replica
I20260808 02:19:18.422724 13720 raft_consensus.cc:2229] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:19:18.422807 13720 raft_consensus.cc:2258] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:19:18.426599 13720 ts_tablet_manager.cc:1905] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:19:18.429317 13716 tablet_replica.cc:330] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: stopping tablet replica
I20260808 02:19:18.429533 13716 raft_consensus.cc:2229] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:19:18.429713 13716 raft_consensus.cc:2258] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:19:18.430480 12979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:43841 (error 108)
I20260808 02:19:18.430979 13720 ts_tablet_manager.cc:1918] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:19:18.431077 13720 log.cc:1196] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/728f1cae91b34aafb72cc2f895cf9864
I20260808 02:19:18.431620 13716 ts_tablet_manager.cc:1905] T 728f1cae91b34aafb72cc2f895cf9864 P ed27b94f95c34bb8a76a7fac99ce64e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:19:18.431753 13720 ts_tablet_manager.cc:1939] T 728f1cae91b34aafb72cc2f895cf9864 P 321c5620e2f94cc4a71d9dcc341ab645: Deleting consensus metadata
I20260808 02:19:18.432242 12976 catalog_manager.cc:4841] TS 321c5620e2f94cc4a71d9dcc341ab645 (127.28.177.131:34677): tablet 728f1cae91b34aafb72cc2f895cf9864 (table test-table [id=70b5f2e74a7849868a4f5114916f6877]) successfully deleted
I20260808 02:19:18.432780 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 13292
I20260808 02:19:18.445449 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:19:18.445236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:43841 (local address 127.0.0.1:40614)
0808 02:19:18.445321 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.445393 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:43841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 02:19:18.445641 12979 catalog_manager.cc:4584] TS 61d115085d05464ba32db496e516f22b (127.28.177.129:43841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 728f1cae91b34aafb72cc2f895cf9864: Network error: Client connection negotiation failed: client connection to 127.28.177.129:43841: connect: Connection refused (error 111)
W20260808 02:19:18.453042 13681 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:19:18.452683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37983 (local address 127.0.0.1:60874)
0808 02:19:18.452747 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 02:19:18.452758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:19:18.452802 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:19:18.452976 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.130:37983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:19:18.453836 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 13490
I20260808 02:19:18.466472 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 12946
I20260808 02:19:18.479508 29382 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:19:18.826550 29382 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:19:18.826700 29382 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:19:19.181888 29382 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:19:19.182 UTC [12369] LOG: received smart shutdown request
2026-08-08 02:19:19.187 UTC [12369] LOG: background worker "logical replication launcher" (PID 12379) exited with exit code 1
2026-08-08 02:19:19.187 UTC [12374] LOG: shutting down
2026-08-08 02:19:19.381 UTC [12369] LOG: database system is shut down
2026-08-08T02:19:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54362 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:19:19 dist-test-slave-3h5h krb5kdc[13732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:19:19 dist-test-slave-3h5h krb5kdc[13732](info): set up 2 sockets
Aug 08 02:19:19 dist-test-slave-3h5h krb5kdc[13732](info): commencing operation
krb5kdc: starting...
W20260808 02:19:20.826861 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:19:20 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:19:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:19:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/rangeradmin_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/rangeradmin_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/rangerlookup_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/rangerlookup_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
I20260808 02:19:21.023525 29382 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfrqfAr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/postgres -l logfile start
2026-08-08 02:19:24.017 UTC [13773] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:19:24.017 UTC [13773] LOG: listening on IPv4 address "127.28.177.148", port 38361
2026-08-08 02:19:24.022 UTC [13773] LOG: listening on Unix socket "/tmp/.s.PGSQL.38361"
2026-08-08 02:19:24.048 UTC [13777] LOG: database system was shut down at 2026-08-08 02:19:22 UTC
2026-08-08 02:19:24.053 UTC [13773] LOG: database system is ready to accept connections
I20260808 02:19:25.283447 29382 mini_postgres.cc:96] Postgres bound to 38361
2026-08-08 02:19:25.291 UTC [13787] FATAL: database "slave" does not exist
127.28.177.148:38361 - accepting connections
I20260808 02:19:25.292038 29382 mini_ranger.cc:161] Starting Ranger...
I20260808 02:19:25.311823 29382 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:19:25.904755 29382 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:19:25.904889 29382 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-admin
2026-08-08 02:19:26,211 [I] DB FLAVOR :POSTGRES
2026-08-08 02:19:26,212 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:19:26,212 [I] Checking connection..
2026-08-08 02:19:26,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:19:26,481 [I] Checking connection passed.
2026-08-08 02:19:26,481 [I] --------- Verifying version history table ---------
2026-08-08 02:19:26,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:19:26,763 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:19:26,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:19:27,060 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:19:27,061 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:19:27,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:19:27.301 UTC [13864] WARNING: there is no transaction in progress
2026-08-08 02:19:27,311 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:19:27,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:19:27,567 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:19:27,568 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:19:27,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:19:27,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:19:28,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3h5h.c.gcp-upstream.internal','N') ;"
2026-08-08 02:19:28,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:19:28,699 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:19:28,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:19:28,997 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:19:28,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:19:29,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:19:29,553 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:19:29,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:19:30.672 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.691 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.709 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.728 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.750 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.769 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.833 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.841 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.859 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.871 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.883 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.900 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.912 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.920 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.929 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:30.936 UTC [14005] WARNING: there is no transaction in progress
2026-08-08 02:19:31,769 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:19:31,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:19:32,036 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:19:32,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:19:32,314 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:19:32,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:19:32,589 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:19:32,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:19:32,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:19:33,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3h5h.c.gcp-upstream.internal';"
2026-08-08 02:19:33,422 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:19:33,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:19:33,695 [I] DB_PATCHES have already been applied
I20260808 02:19:33.701116 29382 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfrqfAr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:19:33.701205 29382 mini_ranger.cc:193] Using host: 127.28.177.148
I20260808 02:19:33.705447 29382 mini_ranger.cc:227] Ranger admin URL: http://127.28.177.148:44277
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:19:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:19:35.111387 29382 mini_ranger.cc:160] Time spent starting Ranger: real 9.819s user 0.000s sys 0.011s
Aug 08, 2026 2:19:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:19:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 2:19:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:19:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:19:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:19:59.912894 29382 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.177.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.177.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/rangerkms_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.177.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/rangerkms_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.177.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.177.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.177.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/HTTP_127.28.177.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:20:00 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155600, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:20:00.018708 29382 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:20:00.030805 29382 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:20:01.126178 29382 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:20:01.126279 29382 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms
I20260808 02:20:01.126304 29382 mini_ranger_kms.cc:227] Using postgres at 127.28.177.148:38361
2026-08-08 02:20:01,321 [I] DB FLAVOR :POSTGRES
2026-08-08 02:20:01,322 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:20:01,322 [I] Checking connection
2026-08-08 02:20:01,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:20:01,568 [I] connection success
2026-08-08 02:20:01,569 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:20:01,569 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:20:01,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:20:01,837 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:20:01,837 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:20:01,837 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:20:01,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.177.148:38361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:20:02,133 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:20:02.715476 29382 mini_ranger_kms.cc:310] Created kms service
I20260808 02:20:02.935868 29382 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:20:02.993424 29382 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:20:02 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155600, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:03.484699 29382 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:20:03.485036 29382 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfrqfAr/build/release/bin/postgresql.jar:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfrqfAr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:20:03.485088 29382 mini_ranger_kms.cc:240] Using host: 127.28.177.148
I20260808 02:20:03.490139 29382 mini_ranger_kms.cc:276] Ranger KMS PID: 14252
I20260808 02:20:03.490221 29382 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.177.148:39081
Aug 08, 2026 2:20:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:20:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:20:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:20:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:20:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:20:05 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155605, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.177.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:06.296010 29382 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.277s user 0.010s sys 0.166s
I20260808 02:20:06.296132 29382 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:20:06.296185 29382 mini_ranger_kms.cc:401] 127.28.177.148:39081/kms/v1/keys
Aug 08 02:20:06 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155605, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.177.148@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
Aug 08 02:20:06 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155605, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.177.148@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
Aug 08 02:20:09 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:20:09.357375 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:35405
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:35405
--ranger_config_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:09.444396 14316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:09.444620 14316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:09.444643 14316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:09.445906 14316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:09.445950 14316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:09.445966 14316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:09.445977 14316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:09.445990 14316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:09.447588 14316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.177.148:39081/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:35405
--ranger_config_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:35405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:09.447841 14316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:09.448099 14316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:09.453783 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:09.453749 14324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:09.454077 14316 server_base.cc:1034] running on GCE node
W20260808 02:20:09.453763 14327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:09.454512 14316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:09.454906 14316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:09.456100 14316 hybrid_clock.cc:648] HybridClock initialized: now 1786155609456079 us; error 36 us; skew 500 ppm
Aug 08 02:20:09 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155609, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:09.464141 14316 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:09.464727 14316 webserver.cc:460] Webserver started at http://127.28.177.190:36057/ using document root <none> and password file <none>
I20260808 02:20:09.465000 14316 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:09.465045 14316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:09.465186 14316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:20:09 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155609, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:09.534255 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "bfd246ffd0d841c89cfde59c2137f311"
format_stamp: "Formatted at 2026-08-08 02:20:09 on dist-test-slave-3h5h"
server_key: "f429b8ed1d0bfda33dfd9d31e2666702"
server_key_iv: "8cf5310f7d4c329bf16dff1ebefb7e41"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:09.534698 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "bfd246ffd0d841c89cfde59c2137f311"
format_stamp: "Formatted at 2026-08-08 02:20:09 on dist-test-slave-3h5h"
server_key: "f429b8ed1d0bfda33dfd9d31e2666702"
server_key_iv: "8cf5310f7d4c329bf16dff1ebefb7e41"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:09.536166 14316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:20:09.552950 14334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:09.553135 14316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:20:09.553231 14316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "bfd246ffd0d841c89cfde59c2137f311"
format_stamp: "Formatted at 2026-08-08 02:20:09 on dist-test-slave-3h5h"
server_key: "f429b8ed1d0bfda33dfd9d31e2666702"
server_key_iv: "8cf5310f7d4c329bf16dff1ebefb7e41"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:09.553328 14316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:09.569221 14316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:09.571277 14316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:09.571455 14316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:09.576761 14316 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:35405
I20260808 02:20:09.576861 14410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:35405 every 8 connection(s)
I20260808 02:20:09.577248 14316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:09.578022 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:09.580454 14412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Bootstrap starting.
I20260808 02:20:09.581200 14412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:09.581513 14412 log.cc:824] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:09.582335 14412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: No bootstrap required, opened a new log
I20260808 02:20:09.584434 14412 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:09.584622 14412 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:09.584694 14412 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd246ffd0d841c89cfde59c2137f311, State: Initialized, Role: FOLLOWER
I20260808 02:20:09.584915 14412 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:09.585011 14412 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:09.585075 14412 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:09.585076 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 14316
I20260808 02:20:09.585147 14412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:09.585186 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:20:09.585367 29382 external_mini_cluster.cc:1428] Setting key de0392c73721d78917d7b71bc84c4d28
I20260808 02:20:09.585858 14412 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:09.586015 14412 leader_election.cc:302] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd246ffd0d841c89cfde59c2137f311; no voters:
I20260808 02:20:09.586223 14412 leader_election.cc:288] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:20:09.586395 14419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:09.586513 14412 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:09.586572 14419 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 LEADER]: Becoming Leader. State: Replica: bfd246ffd0d841c89cfde59c2137f311, State: Running, Role: LEADER
I20260808 02:20:09.586745 14419 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:09.587129 14419 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfd246ffd0d841c89cfde59c2137f311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } } }
I20260808 02:20:09.587296 14419 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:09.587481 14419 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd246ffd0d841c89cfde59c2137f311. Latest consensus state: current_term: 1 leader_uuid: "bfd246ffd0d841c89cfde59c2137f311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } } }
I20260808 02:20:09.587589 14419 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:09.587801 14430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:09.588165 14412 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:20:09.588392 14430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 02:20:09 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:09.597002 14430 catalog_manager.cc:1347] Generated new cluster ID: ab8828a2f0804704950030c8aef2b184
I20260808 02:20:09.600018 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:09.585938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.28.177.190:35405)
0808 02:20:09.586197 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:09.586207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:09.586242 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:20:09.586367 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:09.586375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:09.586434 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:09.587398 (+ 964us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:09.587517 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:09.588970 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:09.589932 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:09.590575 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:09.590687 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:09.594070 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:09.594122 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:09.594122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:09.594217 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:09.597352 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:09.597773 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:09.597776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:09.597777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:09.597904 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:09.598123 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:09.598124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:09.598124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:09.598476 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:09.599136 (+ 660us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:09.599606 (+ 470us) server_negotiation.cc:299] Negotiation successful
0808 02:20:09.599771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
I20260808 02:20:09.601341 14430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:09.608771 14430 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:20:09.609563 14430 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:09.621519 14430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Generated new TSK 0
I20260808 02:20:09.621834 14430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:20:10.452764 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:35405
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:10.569579 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:10.569797 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:10.569823 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:10.571029 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:10.571074 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:10.571102 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:20:10.572680 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.177.148:39081/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:35405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:10.572960 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:10.573356 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:10.581640 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:10.581979 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:10.581987 14470 server_base.cc:1034] running on GCE node
W20260808 02:20:10.582278 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:10.582571 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:10.582883 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:10.584067 14470 hybrid_clock.cc:648] HybridClock initialized: now 1786155610584037 us; error 44 us; skew 500 ppm
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:10.587695 14470 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:20:10.588414 14470 webserver.cc:460] Webserver started at http://127.28.177.129:34895/ using document root <none> and password file <none>
I20260808 02:20:10.588785 14470 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:10.588900 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:10.589071 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:10.604315 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "4414b75d99c14874b4413cd36fa78782"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "170044ab68562bffcc5879ebbf78eca3"
server_key_iv: "037b0661870cd06eed84d30d459636e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:10.605424 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "4414b75d99c14874b4413cd36fa78782"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "170044ab68562bffcc5879ebbf78eca3"
server_key_iv: "037b0661870cd06eed84d30d459636e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:10.614253 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.002s
I20260808 02:20:10.630529 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:10.630828 14470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260808 02:20:10.630931 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "4414b75d99c14874b4413cd36fa78782"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "170044ab68562bffcc5879ebbf78eca3"
server_key_iv: "037b0661870cd06eed84d30d459636e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:10.631024 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:10.657887 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:10.660459 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:10.660670 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:10.660957 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:10.661387 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:10.661471 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:10.661532 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:10.661579 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:10.668180 14470 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:40057
I20260808 02:20:10.668677 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:20:10.672197 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:40057 every 8 connection(s)
I20260808 02:20:10.677660 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 14470
I20260808 02:20:10.677769 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:20:10.677981 29382 external_mini_cluster.cc:1428] Setting key 3d2a6e81427c01d5e67253c19552c689
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:10.684012 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:10.671204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35405 (local address 127.28.177.129:33391)
0808 02:20:10.672512 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:10.672556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:10.673261 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:10.673727 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:10.673732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:10.673999 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:10.674308 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:10.674314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.675383 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.675384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:10.676697 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:10.676701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.677304 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.677305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:10.677458 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:10.678265 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:10.678277 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:10.680015 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:10.682289 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:10.682298 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:10.682301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:10.682605 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:10.682833 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:10.682834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:10.682834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:10.682968 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:10.683285 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:10.683291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:10.683470 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 02:20:10.683586 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 02:20:10.683758 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"thread_start_us":586,"threads_started":1}
I20260808 02:20:10.684608 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:10.672028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:33391 (local address 127.28.177.190:35405)
0808 02:20:10.673462 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:10.673464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:10.673473 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:20:10.673501 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:10.673502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:10.673515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:10.673577 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:10.674422 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.675266 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.676800 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.677151 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.677227 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:10.680136 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:10.680138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:10.680138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:10.680148 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:10.682182 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:10.682693 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:10.682694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:10.682695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:10.682761 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:10.683040 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:10.683041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:10.683041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:10.683193 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:10.683256 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:10.684407 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0808 02:20:10.684501 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":62,"threads_started":1}
I20260808 02:20:10.685487 14647 heartbeater.cc:347] Connected to a master server at 127.28.177.190:35405
I20260808 02:20:10.685593 14647 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:10.685799 14647 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:10.686491 14358 ts_manager.cc:194] Registered new tserver with Master: 4414b75d99c14874b4413cd36fa78782 (127.28.177.129:40057)
I20260808 02:20:10.687366 14358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:33391
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:20:10.722669 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:35405
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:20:10.864384 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:10.864789 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:10.864902 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:10.866992 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:10.867105 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:10.867206 14658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:20:10.869771 14658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.177.148:39081/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:35405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:10.870368 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:10.870745 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:10.877947 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:10.877965 14669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:10.878408 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:10.879563 14658 server_base.cc:1034] running on GCE node
I20260808 02:20:10.879832 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:10.880177 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:10.881367 14658 hybrid_clock.cc:648] HybridClock initialized: now 1786155610881351 us; error 35 us; skew 500 ppm
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:10.884933 14658 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:20:10.885462 14658 webserver.cc:460] Webserver started at http://127.28.177.130:43271/ using document root <none> and password file <none>
I20260808 02:20:10.885797 14658 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:10.885891 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:10.886070 14658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:10.902487 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "bf04bb725fc28249090dd8ce15807544"
server_key_iv: "a6a895376aad64e055beb9f63486533f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:10.903038 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "bf04bb725fc28249090dd8ce15807544"
server_key_iv: "a6a895376aad64e055beb9f63486533f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:10.905221 14658 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:20:10.941260 14684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:10.941679 14658 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260808 02:20:10.941874 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "bf04bb725fc28249090dd8ce15807544"
server_key_iv: "a6a895376aad64e055beb9f63486533f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:10.942082 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:10.954986 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:10.957866 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:10.958153 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:10.958521 14658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:10.958992 14658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:10.959190 14658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:10.959280 14658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:10.959345 14658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:10.980262 14658 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:35215
I20260808 02:20:10.980952 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:20:10.981662 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:35215 every 8 connection(s)
I20260808 02:20:10.990295 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 14658
I20260808 02:20:10.990402 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:20:10.990621 29382 external_mini_cluster.cc:1428] Setting key 952e915875e8a8632327f2e43faa5f6e
Aug 08 02:20:10 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:10.995590 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:10.983277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:59309 (local address 127.28.177.190:35405)
0808 02:20:10.983338 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:10.983340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:10.983687 (+ 347us) server_negotiation.cc:407] Connection header received
0808 02:20:10.984530 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:10.984531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:10.984551 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:10.984589 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:10.985712 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.986535 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.988203 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.988574 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.988627 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:10.991889 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:10.991890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:10.991891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:10.991900 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:10.993781 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:10.994333 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:10.994334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:10.994335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:10.994405 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:10.994734 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:10.994735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:10.994735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:10.994881 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:10.994964 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:10.995325 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 02:20:10.995481 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:20:10.995750 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:10.982953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35405 (local address 127.28.177.130:59309)
0808 02:20:10.983539 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:10.983586 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:10.984378 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:10.984803 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:10.984808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:10.985202 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:10.985577 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:10.985584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.986678 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.986679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:10.988082 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:10.988086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:10.988765 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:10.988766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:10.989016 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:10.989737 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:10.989749 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:10.991767 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:10.993878 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:10.993885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:10.993888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:10.994228 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:10.994484 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:10.994485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:10.994485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:10.994647 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:10.994953 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:10.994959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:10.995139 (+ 180us) client_negotiation.cc:769] Sending connection context
0808 02:20:10.995276 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 02:20:10.995465 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":145,"threads_started":1}
I20260808 02:20:10.996496 14844 heartbeater.cc:347] Connected to a master server at 127.28.177.190:35405
I20260808 02:20:10.996613 14844 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:10.996824 14844 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:10.997362 14358 ts_manager.cc:194] Registered new tserver with Master: 1fc2efd7aa904d8c8eb7bcb0015bdad1 (127.28.177.130:35215)
I20260808 02:20:10.997866 14358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:59309
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:20:11.044183 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:35405
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:11.204170 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:11.204484 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:11.204571 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:11.205994 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:11.206130 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:11.206197 14850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:20:11.208094 14850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.177.148:39081/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:35405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:11.208388 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:11.208709 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:11.214205 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:11.214600 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:11.215413 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:11.215708 14850 server_base.cc:1034] running on GCE node
I20260808 02:20:11.215920 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:11.216214 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:11.217387 14850 hybrid_clock.cc:648] HybridClock initialized: now 1786155611217365 us; error 48 us; skew 500 ppm
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:11.220280 14850 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:20:11.220777 14850 webserver.cc:460] Webserver started at http://127.28.177.131:35045/ using document root <none> and password file <none>
I20260808 02:20:11.221125 14850 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:11.221200 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:11.221375 14850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:11.242844 14850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "657f47fe115d4234995247420fbb4c8f"
format_stamp: "Formatted at 2026-08-08 02:20:11 on dist-test-slave-3h5h"
server_key: "4d56f41e938af7533ae0bbf7233d74dd"
server_key_iv: "8cd7a9aca224943eb54190ac326e255f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:11.243338 14850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "657f47fe115d4234995247420fbb4c8f"
format_stamp: "Formatted at 2026-08-08 02:20:11 on dist-test-slave-3h5h"
server_key: "4d56f41e938af7533ae0bbf7233d74dd"
server_key_iv: "8cd7a9aca224943eb54190ac326e255f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:11.247864 14850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:20:11.265781 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.266081 14850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 02:20:11.266197 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "657f47fe115d4234995247420fbb4c8f"
format_stamp: "Formatted at 2026-08-08 02:20:11 on dist-test-slave-3h5h"
server_key: "4d56f41e938af7533ae0bbf7233d74dd"
server_key_iv: "8cd7a9aca224943eb54190ac326e255f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:11.266304 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:11.280596 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:11.283360 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:11.283599 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:11.283888 14850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:11.284358 14850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:11.284458 14850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.284538 14850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:11.284601 14850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.292053 14850 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:44373
I20260808 02:20:11.293051 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:20:11.295334 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 14850
I20260808 02:20:11.295439 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:20:11.295665 29382 external_mini_cluster.cc:1428] Setting key 677cde34b9a0dd7910ca91dd09175ef7
I20260808 02:20:11.303373 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:44373 every 8 connection(s)
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:11.317018 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:11.305146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:37089 (local address 127.28.177.190:35405)
0808 02:20:11.305202 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:11.305204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:11.305602 (+ 398us) server_negotiation.cc:407] Connection header received
0808 02:20:11.306331 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:11.306332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:11.306347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:11.306407 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:11.307867 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.308655 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.310321 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.310630 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.310670 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:11.313589 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:11.313590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:11.313591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:11.313600 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:11.315402 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:11.315907 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:11.315909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:11.315909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:11.315969 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:11.316254 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:11.316254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:11.316255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:11.316390 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:11.316441 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:11.316796 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 02:20:11.316915 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:20:11.318187 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:11.304073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35405 (local address 127.28.177.131:37089)
0808 02:20:11.305452 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:11.305504 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:11.306207 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:11.306534 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:11.306539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:11.307454 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:11.307753 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:11.307760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.308770 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.308771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:11.310213 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:11.310216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.310779 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.310780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:11.310919 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:11.311543 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:11.311553 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:11.313470 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:11.315498 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:11.315504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:11.315507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:11.315812 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:11.316035 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:11.316036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:11.316036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:11.316179 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:11.316472 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:11.316477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:11.316642 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 02:20:11.317125 (+ 483us) client_negotiation.cc:240] Negotiation successful
0808 02:20:11.317920 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":179,"threads_started":1}
I20260808 02:20:11.321009 15016 heartbeater.cc:347] Connected to a master server at 127.28.177.190:35405
I20260808 02:20:11.321132 15016 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:11.321354 15016 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:11.321785 14358 ts_manager.cc:194] Registered new tserver with Master: 657f47fe115d4234995247420fbb4c8f (127.28.177.131:44373)
I20260808 02:20:11.322299 14358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:37089
I20260808 02:20:11.330025 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:20:11.330837 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 14470
I20260808 02:20:11.348074 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 14658
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155610, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:11.360922 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 14850
I20260808 02:20:11.392943 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 14316
I20260808 02:20:11.407217 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:35405
--webserver_interface=127.28.177.190
--webserver_port=36057
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:35405
--ranger_config_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:11.501155 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:11.501405 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:11.501432 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:11.502692 15019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:11.502733 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:11.502749 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:11.502763 15019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:11.502777 15019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:11.505364 15019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:35405
--ranger_config_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:35405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=36057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:11.505609 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:11.505803 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:20:11.512130 15019 server_base.cc:1034] running on GCE node
W20260808 02:20:11.512148 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:11.512308 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:11.512176 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:11.512666 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:11.513068 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:11.514282 15019 hybrid_clock.cc:648] HybridClock initialized: now 1786155611514240 us; error 42 us; skew 500 ppm
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:11.517153 15019 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:11.517732 15019 webserver.cc:460] Webserver started at http://127.28.177.190:36057/ using document root <none> and password file <none>
I20260808 02:20:11.518023 15019 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:11.518107 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:11.566860 15019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:11.567714 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.568148 15019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:11.568275 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "bfd246ffd0d841c89cfde59c2137f311"
format_stamp: "Formatted at 2026-08-08 02:20:09 on dist-test-slave-3h5h"
server_key: "f429b8ed1d0bfda33dfd9d31e2666702"
server_key_iv: "8cf5310f7d4c329bf16dff1ebefb7e41"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:11.568671 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:11.591174 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:11.593569 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:11.593815 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:11.598917 15019 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:35405
I20260808 02:20:11.598968 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:35405 every 8 connection(s)
I20260808 02:20:11.599398 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:11.600757 15105 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:20:11.601471 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Bootstrap starting.
I20260808 02:20:11.604724 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 15019
I20260808 02:20:11.605139 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:40057
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:35405
--webserver_port=34895
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:11.610608 15105 log.cc:824] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:11.612542 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:20:11.612936 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Bootstrap complete.
I20260808 02:20:11.615444 15105 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:11.615727 15105 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd246ffd0d841c89cfde59c2137f311, State: Initialized, Role: FOLLOWER
I20260808 02:20:11.615958 15105 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:11.616060 15105 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:11.616115 15105 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:11.616178 15105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:20:11.617321 15105 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:11.617447 15105 leader_election.cc:302] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd246ffd0d841c89cfde59c2137f311; no voters:
I20260808 02:20:11.617663 15105 leader_election.cc:288] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:20:11.617702 15109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:20:11.617812 15109 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [term 2 LEADER]: Becoming Leader. State: Replica: bfd246ffd0d841c89cfde59c2137f311, State: Running, Role: LEADER
I20260808 02:20:11.617913 15109 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } }
I20260808 02:20:11.617959 15105 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:11.618216 15111 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bfd246ffd0d841c89cfde59c2137f311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } } }
I20260808 02:20:11.618309 15111 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:11.618211 15109 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd246ffd0d841c89cfde59c2137f311. Latest consensus state: current_term: 2 leader_uuid: "bfd246ffd0d841c89cfde59c2137f311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd246ffd0d841c89cfde59c2137f311" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 35405 } } }
I20260808 02:20:11.618508 15109 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:11.618724 15118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:11.619288 15118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:11.619436 15118 catalog_manager.cc:1259] Loaded cluster ID: ab8828a2f0804704950030c8aef2b184
I20260808 02:20:11.619481 15118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:11.619637 15105 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:20:11.623978 15118 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:11.624629 15118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Loaded TSK: 0
I20260808 02:20:11.625010 15118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:20:11.736508 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:11.736809 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:11.736884 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:11.738761 15107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:11.738838 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:11.738886 15107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:20:11.741417 15107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:40057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=34895
--tserver_master_addrs=127.28.177.190:35405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:11.741714 15107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:11.742038 15107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:11.749670 15165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:11.749697 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:11.750473 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:11.750782 15107 server_base.cc:1034] running on GCE node
I20260808 02:20:11.751006 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:11.751312 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:11.754586 15107 hybrid_clock.cc:648] HybridClock initialized: now 1786155611754549 us; error 48 us; skew 500 ppm
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:11.763461 15107 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:20:11.764093 15107 webserver.cc:460] Webserver started at http://127.28.177.129:34895/ using document root <none> and password file <none>
I20260808 02:20:11.764405 15107 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:11.764501 15107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:11.799510 15107 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.001s
I20260808 02:20:11.800999 15178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.801267 15107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:20:11.801393 15107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "4414b75d99c14874b4413cd36fa78782"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "170044ab68562bffcc5879ebbf78eca3"
server_key_iv: "037b0661870cd06eed84d30d459636e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:11.801904 15107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:11.837312 15107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:11.840293 15107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:11.840476 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:11.840734 15107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:11.841159 15107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:11.841215 15107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.841254 15107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:11.841280 15107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:11.848119 15107 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:40057
I20260808 02:20:11.848616 15107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:20:11.858024 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 15107
I20260808 02:20:11.858388 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:35215
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:35405
--webserver_port=43271
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:11.874938 15339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:40057 every 8 connection(s)
Aug 08 02:20:11 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155611, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:11.893790 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:11.875859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35405 (local address 127.28.177.129:56333)
0808 02:20:11.877291 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:11.877345 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:11.878055 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:11.878808 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:11.878813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:11.879102 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:11.879359 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:11.879365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.880935 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.880937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:11.882398 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:11.882403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.883393 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.883395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:11.883545 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:11.884331 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:11.884343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:11.886051 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:11.891251 (+ 5200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:11.891262 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:11.891266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:11.891657 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:11.892002 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:11.892003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:11.892004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:11.892195 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:11.892893 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:11.892901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:11.893096 (+ 195us) client_negotiation.cc:769] Sending connection context
0808 02:20:11.893223 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 02:20:11.893426 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":137,"threads_started":1}
I20260808 02:20:11.894426 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:11.875906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:56333 (local address 127.28.177.190:35405)
0808 02:20:11.877189 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:11.877202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:11.878250 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 02:20:11.878423 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:11.878430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:11.878504 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:11.878644 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:11.879537 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.880777 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.882517 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:11.883253 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:11.883402 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:11.886229 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:11.886236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:11.886236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:11.886253 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:11.891113 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:11.891772 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:11.891776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:11.891777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:11.891908 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:11.892291 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:11.892293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:11.892293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:11.892748 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:11.893958 (+ 1210us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:11.894061 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:20:11.894213 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":93,"threads_started":1}
I20260808 02:20:11.895126 15340 heartbeater.cc:347] Connected to a master server at 127.28.177.190:35405
I20260808 02:20:11.895275 15340 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:11.895493 15340 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
W20260808 02:20:11.896514 15340 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:35405 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:20:11.896617 15340 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:11.896706 15340 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:11.897104 15340 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:11.897194 15340 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
W20260808 02:20:11.897527 15340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:20:12.012002 15342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:12.012347 15342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:12.012399 15342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:12.014552 15342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:12.014626 15342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:12.014675 15342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:20:12.017388 15342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:35215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=43271
--tserver_master_addrs=127.28.177.190:35405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:12.017714 15342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:12.018034 15342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:12.024740 15354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:12.024719 15355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:12.024744 15357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:12.025295 15342 server_base.cc:1034] running on GCE node
I20260808 02:20:12.025565 15342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:12.025898 15342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:12.027113 15342 hybrid_clock.cc:648] HybridClock initialized: now 1786155612027070 us; error 50 us; skew 500 ppm
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:12.030874 15342 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:20:12.031473 15342 webserver.cc:460] Webserver started at http://127.28.177.130:43271/ using document root <none> and password file <none>
I20260808 02:20:12.031775 15342 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:12.031877 15342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:12.056289 15342 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260808 02:20:12.057650 15368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:12.057963 15342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:12.058086 15342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1"
format_stamp: "Formatted at 2026-08-08 02:20:10 on dist-test-slave-3h5h"
server_key: "bf04bb725fc28249090dd8ce15807544"
server_key_iv: "a6a895376aad64e055beb9f63486533f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:12.058645 15342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:12.081013 15342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:12.083559 15342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:12.083755 15342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:12.084012 15342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:12.084380 15342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:12.084438 15342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:12.084473 15342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:12.084517 15342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:12.093048 15342 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:35215
I20260808 02:20:12.093497 15342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:20:12.100726 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 15342
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:12.101048 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.177.148:39081/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:44373
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:35405
--webserver_port=35045
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:12.105544 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:12.094387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:45175 (local address 127.28.177.190:35405)
0808 02:20:12.094443 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:12.094446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:12.094693 (+ 247us) server_negotiation.cc:407] Connection header received
0808 02:20:12.095272 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:12.095273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:12.095289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:12.095320 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:12.096136 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.096962 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.098438 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.098665 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.098760 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:12.101722 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:12.101725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:12.101725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:12.101734 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:12.103640 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:12.104239 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:12.104241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:12.104241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:12.104311 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:12.104642 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:12.104644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:12.104644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:12.104805 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:12.104889 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:12.105306 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 02:20:12.105412 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:20:12.106164 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:12.094376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35405 (local address 127.28.177.130:45175)
0808 02:20:12.094565 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:12.094610 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:12.095118 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:12.095488 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:12.095491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:12.095718 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:12.096013 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:12.096020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.097094 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.097095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:12.098337 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:12.098340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.098793 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.098795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:12.098924 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:12.099533 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:12.099544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:12.101566 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:12.103743 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:12.103751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:12.103753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:12.104112 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:12.104391 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:12.104392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:12.104393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:12.104560 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:12.104942 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:12.104948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:12.105136 (+ 188us) client_negotiation.cc:769] Sending connection context
0808 02:20:12.105705 (+ 569us) client_negotiation.cc:240] Negotiation successful
0808 02:20:12.105890 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260808 02:20:12.106231 15529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:35215 every 8 connection(s)
I20260808 02:20:12.106797 15530 heartbeater.cc:347] Connected to a master server at 127.28.177.190:35405
I20260808 02:20:12.106921 15530 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.107170 15530 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
W20260808 02:20:12.107931 15530 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:35405 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:20:12.108032 15530 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.108109 15530 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:12.108467 15530 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.108563 15530 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
W20260808 02:20:12.108925 15530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:20:12.222170 15533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:12.222400 15533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:12.222450 15533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:12.223636 15533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:12.223704 15533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:12.223752 15533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:20:12.225323 15533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.177.148:39081/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:44373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=35045
--tserver_master_addrs=127.28.177.190:35405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:12.225596 15533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:12.225837 15533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:12.231638 15543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:12.231712 15547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:12.231719 15545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:12.232246 15533 server_base.cc:1034] running on GCE node
I20260808 02:20:12.232498 15533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:12.232796 15533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:12.234033 15533 hybrid_clock.cc:648] HybridClock initialized: now 1786155612234010 us; error 31 us; skew 500 ppm
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:12.237147 15533 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:20:12.237738 15533 webserver.cc:460] Webserver started at http://127.28.177.131:35045/ using document root <none> and password file <none>
I20260808 02:20:12.237977 15533 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:12.238065 15533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:12.257485 15533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:20:12.258788 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:12.259066 15533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:12.259191 15533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "657f47fe115d4234995247420fbb4c8f"
format_stamp: "Formatted at 2026-08-08 02:20:11 on dist-test-slave-3h5h"
server_key: "4d56f41e938af7533ae0bbf7233d74dd"
server_key_iv: "8cd7a9aca224943eb54190ac326e255f"
server_key_version: "kuduclusterkey@0"
I20260808 02:20:12.259632 15533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:12.280205 15533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:12.283097 15533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:12.283319 15533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:12.283643 15533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:12.284083 15533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:12.284133 15533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:12.284169 15533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:12.284195 15533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:12.293088 15533 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:44373
I20260808 02:20:12.293387 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:44373 every 8 connection(s)
I20260808 02:20:12.293579 15533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:12.302158 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 15533
I20260808 02:20:12.305025 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:12.294506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:57899 (local address 127.28.177.190:35405)
0808 02:20:12.294560 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:12.294564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:12.294816 (+ 252us) server_negotiation.cc:407] Connection header received
0808 02:20:12.295521 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:12.295523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:12.295537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:12.295570 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:12.296473 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.297061 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.298103 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.298328 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.298410 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:12.300889 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:12.300891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:12.300892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:12.300901 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:12.302706 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:12.303782 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:12.303783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:12.303784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:12.303850 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:12.304150 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:12.304151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:12.304152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:12.304296 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:12.304340 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:12.304777 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 02:20:12.304914 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:20:12.305801 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:12.294517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:35405 (local address 127.28.177.131:57899)
0808 02:20:12.294692 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:12.294736 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:12.295376 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:12.295763 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:12.295768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:12.296121 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:12.296355 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:12.296360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.297162 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.297163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:12.297999 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:12.298001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.298438 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.298439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:12.298571 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:12.299154 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:12.299163 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:12.300731 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:12.303341 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:12.303350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:12.303353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:12.303681 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:12.303922 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:12.303923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:12.303923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:12.304073 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:12.304388 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:12.304395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:12.304612 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 02:20:12.305385 (+ 773us) client_negotiation.cc:240] Negotiation successful
0808 02:20:12.305564 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260808 02:20:12.306552 15728 heartbeater.cc:347] Connected to a master server at 127.28.177.190:35405
I20260808 02:20:12.306680 15728 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.306885 15728 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
W20260808 02:20:12.307659 15728 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:35405 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:20:12.307744 15728 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.307825 15728 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:12.308228 15728 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.308332 15728 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
W20260808 02:20:12.308754 15728 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:20:12.315938 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:12.303252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.28.177.190:35405)
0808 02:20:12.305100 (+ 1848us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:12.305102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:12.309064 (+ 3962us) server_negotiation.cc:407] Connection header received
0808 02:20:12.309144 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:12.309145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:12.309158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:12.309226 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:12.309893 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.310383 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.311195 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:12.311417 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:12.311507 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:12.312967 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:12.312969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:12.312970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:12.312979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:12.314602 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:12.315033 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:12.315035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:12.315035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:12.315102 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:12.315340 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:12.315341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:12.315341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:12.315471 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:12.315685 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:12.315726 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:20:12.315810 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1833,"thread_start_us":50,"threads_started":1}
Aug 08 02:20:12 dist-test-slave-3h5h krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:12.897781 15340 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:12.897971 15340 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:12.898609 15059 ts_manager.cc:194] Registered new tserver with Master: 4414b75d99c14874b4413cd36fa78782 (127.28.177.129:40057)
I20260808 02:20:12.899657 15059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:56333
I20260808 02:20:13.109119 15530 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:13.109295 15530 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:13.109647 15059 ts_manager.cc:194] Registered new tserver with Master: 1fc2efd7aa904d8c8eb7bcb0015bdad1 (127.28.177.130:35215)
I20260808 02:20:13.110205 15059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:45175
Aug 08 02:20:13 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
I20260808 02:20:13.308921 15728 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:13.309121 15728 heartbeater.cc:511] Master 127.28.177.190:35405 requested a full tablet report, sending...
I20260808 02:20:13.309414 15059 ts_manager.cc:194] Registered new tserver with Master: 657f47fe115d4234995247420fbb4c8f (127.28.177.131:44373)
I20260808 02:20:13.309890 15059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:57899
I20260808 02:20:13.319201 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:20:13.327178 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.320336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.28.177.190:35405)
0808 02:20:13.320605 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.320607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.320613 (+ 6us) server_negotiation.cc:407] Connection header received
0808 02:20:13.320632 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.320632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.320642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.320718 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:13.321345 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.321924 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.322705 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.322917 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.322994 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.324276 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:13.324278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:13.324279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:13.324287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:13.325875 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:13.326256 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:13.326258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:13.326258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:13.326324 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:13.326529 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:13.326530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:13.326530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:13.326652 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:13.326763 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.326875 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.326963 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":70,"threads_started":1}
I20260808 02:20:13.329244 15059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:20:13.331089 15059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:20:13.345748 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.334004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:35215 (local address 127.0.0.1:45454)
0808 02:20:13.340983 (+ 6979us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:13.341028 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:13.341134 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:13.341455 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:13.341458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:13.341460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:13.341539 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.341541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.342964 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.342965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.344318 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.344321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.345397 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.345398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.345487 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.345505 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:20:13.345575 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:20:13.345654 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6931,"thread_start_us":125,"threads_started":1}
I20260808 02:20:13.346150 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.334398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.28.177.129:40057)
0808 02:20:13.334874 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.334879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.341356 (+ 6477us) server_negotiation.cc:407] Connection header received
0808 02:20:13.341398 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.341400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.341417 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.341490 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:13.341707 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.343181 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.344426 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.345209 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.345299 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.345357 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.346013 (+ 656us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.346058 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":156,"threads_started":1}
I20260808 02:20:13.345947 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.334288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40057 (local address 127.0.0.1:38180)
0808 02:20:13.341230 (+ 6942us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:13.341239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:13.341289 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:13.341577 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:13.341578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:13.341580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:13.341629 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.341630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.343270 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.343271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.344325 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.344327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.345702 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.345703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.345772 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.345777 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:20:13.345827 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:20:13.345872 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6925,"thread_start_us":73,"threads_started":1}
I20260808 02:20:13.347153 15228 tablet_service.cc:1467] Processing CreateTablet for tablet e91dc8f745f44c67a3e97edfc2711f43 (DEFAULT_TABLE table=test-table [id=0f698e6b7376413dbd2a9d3a244161de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:13.347558 15228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91dc8f745f44c67a3e97edfc2711f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:13.350927 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.334004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44373 (local address 127.0.0.1:51328)
0808 02:20:13.340980 (+ 6976us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:13.341023 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:13.341086 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:13.346384 (+ 5298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:13.346386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:13.346389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:13.346449 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.346452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.347979 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.347980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.349325 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.349329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.350630 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.350631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.350708 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.350714 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:20:13.350776 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:20:13.350824 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6937,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
I20260808 02:20:13.351312 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.334674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.28.177.131:44373)
0808 02:20:13.346023 (+ 11349us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.346029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.346045 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:20:13.346089 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.346092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.346113 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.346223 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:13.346556 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.347846 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.349470 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.350415 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.350520 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.350583 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.351147 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.351201 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11298,"thread_start_us":55,"threads_started":1}
I20260808 02:20:13.346402 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.334144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.28.177.130:35215)
0808 02:20:13.340989 (+ 6845us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.340996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.341140 (+ 144us) server_negotiation.cc:407] Connection header received
0808 02:20:13.341231 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.341234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.341257 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.341341 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:13.341648 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.342867 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.344426 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.345230 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.345299 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.345357 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.346094 (+ 737us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.346137 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6787,"thread_start_us":79,"threads_started":1}
I20260808 02:20:13.352156 15636 tablet_service.cc:1467] Processing CreateTablet for tablet e91dc8f745f44c67a3e97edfc2711f43 (DEFAULT_TABLE table=test-table [id=0f698e6b7376413dbd2a9d3a244161de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:13.352584 15636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91dc8f745f44c67a3e97edfc2711f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:13.357093 15775 tablet_bootstrap.cc:492] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Bootstrap starting.
I20260808 02:20:13.357666 15434 tablet_service.cc:1467] Processing CreateTablet for tablet e91dc8f745f44c67a3e97edfc2711f43 (DEFAULT_TABLE table=test-table [id=0f698e6b7376413dbd2a9d3a244161de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:13.358098 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91dc8f745f44c67a3e97edfc2711f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:13.358665 15774 tablet_bootstrap.cc:492] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: Bootstrap starting.
I20260808 02:20:13.358914 15775 tablet_bootstrap.cc:654] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:13.359629 15774 tablet_bootstrap.cc:654] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:13.360038 15775 log.cc:824] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:13.360944 15774 log.cc:824] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:13.362175 15775 tablet_bootstrap.cc:492] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: No bootstrap required, opened a new log
I20260808 02:20:13.362252 15775 ts_tablet_manager.cc:1397] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 02:20:13.363122 15781 tablet_bootstrap.cc:492] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Bootstrap starting.
I20260808 02:20:13.364409 15781 tablet_bootstrap.cc:654] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:13.365018 15781 log.cc:824] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:13.365968 15781 tablet_bootstrap.cc:492] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: No bootstrap required, opened a new log
I20260808 02:20:13.366056 15781 ts_tablet_manager.cc:1397] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:20:13.366300 15775 raft_consensus.cc:348] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.366588 15775 raft_consensus.cc:374] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:13.366698 15775 raft_consensus.cc:729] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657f47fe115d4234995247420fbb4c8f, State: Initialized, Role: FOLLOWER
I20260808 02:20:13.366902 15775 consensus_queue.cc:260] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.367316 15775 ts_tablet_manager.cc:1428] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 02:20:13.367493 15728 heartbeater.cc:503] Master 127.28.177.190:35405 was elected leader, sending a full tablet report...
I20260808 02:20:13.369302 15781 raft_consensus.cc:348] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.369473 15781 raft_consensus.cc:374] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:13.369515 15781 raft_consensus.cc:729] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc2efd7aa904d8c8eb7bcb0015bdad1, State: Initialized, Role: FOLLOWER
I20260808 02:20:13.369635 15774 tablet_bootstrap.cc:492] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: No bootstrap required, opened a new log
I20260808 02:20:13.369707 15774 ts_tablet_manager.cc:1397] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260808 02:20:13.369707 15781 consensus_queue.cc:260] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.369966 15781 ts_tablet_manager.cc:1428] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 02:20:13.370075 15530 heartbeater.cc:503] Master 127.28.177.190:35405 was elected leader, sending a full tablet report...
I20260808 02:20:13.372435 15774 raft_consensus.cc:348] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.372632 15774 raft_consensus.cc:374] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:13.372663 15774 raft_consensus.cc:729] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4414b75d99c14874b4413cd36fa78782, State: Initialized, Role: FOLLOWER
I20260808 02:20:13.373010 15774 consensus_queue.cc:260] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.373355 15774 ts_tablet_manager.cc:1428] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 02:20:13.373507 15340 heartbeater.cc:503] Master 127.28.177.190:35405 was elected leader, sending a full tablet report...
W20260808 02:20:13.378011 15344 tablet.cc:2369] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:20:13.486845 15785 raft_consensus.cc:482] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:20:13.487236 15785 raft_consensus.cc:504] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.488871 15785 leader_election.cc:288] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 657f47fe115d4234995247420fbb4c8f (127.28.177.131:44373), 4414b75d99c14874b4413cd36fa78782 (127.28.177.129:40057)
I20260808 02:20:13.493733 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.488319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:50243 (local address 127.28.177.131:44373)
0808 02:20:13.488375 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.488377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.489197 (+ 820us) server_negotiation.cc:407] Connection header received
0808 02:20:13.489305 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.489306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.489320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.489374 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:13.489828 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.490765 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.492594 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.493208 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.493269 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.493278 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.493586 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.493637 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 02:20:13.494351 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.488212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:44373 (local address 127.28.177.130:50243)
0808 02:20:13.489085 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:13.489119 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:13.489221 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:13.489508 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:13.489509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:13.489510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:13.489704 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.489707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.490880 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.490881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.492455 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.492458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.493348 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.493349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.493489 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.493493 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:20:13.494041 (+ 548us) client_negotiation.cc:240] Negotiation successful
0808 02:20:13.494088 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":131,"threads_started":1}
I20260808 02:20:13.495745 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.488905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:41125 (local address 127.28.177.129:40057)
0808 02:20:13.488950 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.488952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.489415 (+ 463us) server_negotiation.cc:407] Connection header received
0808 02:20:13.489444 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.489445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.489454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.489502 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:13.489918 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.490641 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.492614 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.495282 (+ 2668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.495334 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.495342 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.495632 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.495680 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:20:13.496009 15663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91dc8f745f44c67a3e97edfc2711f43" candidate_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "657f47fe115d4234995247420fbb4c8f" is_pre_election: true
I20260808 02:20:13.496219 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.488720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40057 (local address 127.28.177.130:41125)
0808 02:20:13.489326 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:13.489334 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:13.489395 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:13.489557 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:13.489558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:13.489561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:13.489809 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.489813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.490755 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.490756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.492501 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:13.492504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.495400 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.495401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:13.495535 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.495539 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:20:13.495971 (+ 432us) client_negotiation.cc:240] Negotiation successful
0808 02:20:13.496014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260808 02:20:13.496243 15663 raft_consensus.cc:2454] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc2efd7aa904d8c8eb7bcb0015bdad1 in term 0.
I20260808 02:20:13.496718 15372 leader_election.cc:302] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc2efd7aa904d8c8eb7bcb0015bdad1, 657f47fe115d4234995247420fbb4c8f; no voters:
I20260808 02:20:13.496898 15785 raft_consensus.cc:2781] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:20:13.496944 15785 raft_consensus.cc:482] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:20:13.496964 15785 raft_consensus.cc:3037] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:13.498298 15785 raft_consensus.cc:504] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.498459 15785 leader_election.cc:288] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [CANDIDATE]: Term 1 election: Requested vote from peers 657f47fe115d4234995247420fbb4c8f (127.28.177.131:44373), 4414b75d99c14874b4413cd36fa78782 (127.28.177.129:40057)
I20260808 02:20:13.499184 15272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91dc8f745f44c67a3e97edfc2711f43" candidate_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4414b75d99c14874b4413cd36fa78782" is_pre_election: true
I20260808 02:20:13.499243 15257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91dc8f745f44c67a3e97edfc2711f43" candidate_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4414b75d99c14874b4413cd36fa78782"
I20260808 02:20:13.499352 15272 raft_consensus.cc:2454] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc2efd7aa904d8c8eb7bcb0015bdad1 in term 0.
I20260808 02:20:13.502048 15663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91dc8f745f44c67a3e97edfc2711f43" candidate_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "657f47fe115d4234995247420fbb4c8f"
I20260808 02:20:13.502171 15663 raft_consensus.cc:3037] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:13.503238 15663 raft_consensus.cc:2454] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc2efd7aa904d8c8eb7bcb0015bdad1 in term 1.
I20260808 02:20:13.503500 15372 leader_election.cc:302] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fc2efd7aa904d8c8eb7bcb0015bdad1, 657f47fe115d4234995247420fbb4c8f; no voters: 4414b75d99c14874b4413cd36fa78782
I20260808 02:20:13.503697 15785 raft_consensus.cc:2781] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:13.503912 15785 raft_consensus.cc:686] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc2efd7aa904d8c8eb7bcb0015bdad1, State: Running, Role: LEADER
I20260808 02:20:13.504061 15785 consensus_queue.cc:237] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } }
I20260808 02:20:13.505255 15059 catalog_manager.cc:5495] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fc2efd7aa904d8c8eb7bcb0015bdad1 (127.28.177.130). New cstate: current_term: 1 leader_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc2efd7aa904d8c8eb7bcb0015bdad1" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 35215 } health_report { overall_health: HEALTHY } } }
I20260808 02:20:13.524060 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.520697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.28.177.130:35215)
0808 02:20:13.520795 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:13.520797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:13.520806 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:20:13.520878 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:13.520879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:13.520903 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:13.520967 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:20:13.521162 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.521894 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.523005 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:13.523283 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:13.523341 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:13.523528 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:20:13.523671 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:20:13.523739 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:13.523957 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 02:20:13.523978 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:20:13.526824 15257 raft_consensus.cc:3037] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:13.527652 15663 raft_consensus.cc:1261] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 1 FOLLOWER]: Refusing update from remote peer 1fc2efd7aa904d8c8eb7bcb0015bdad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:13.528117 15785 consensus_queue.cc:1035] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "657f47fe115d4234995247420fbb4c8f" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:13.528266 15257 raft_consensus.cc:1261] T e91dc8f745f44c67a3e97edfc2711f43 P 4414b75d99c14874b4413cd36fa78782 [term 1 FOLLOWER]: Refusing update from remote peer 1fc2efd7aa904d8c8eb7bcb0015bdad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:13.528568 15785 consensus_queue.cc:1035] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4414b75d99c14874b4413cd36fa78782" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:13.533605 15795 mvcc.cc:205] Tried to move back new op lower bound from 7316093393003335680 to 7316093392913096704. Current Snapshot: MvccSnapshot[applied={T|T < 7316093393003335680}]
I20260808 02:20:13.534289 15797 mvcc.cc:205] Tried to move back new op lower bound from 7316093393003335680 to 7316093392913096704. Current Snapshot: MvccSnapshot[applied={T|T < 7316093393003335680}]
I20260808 02:20:13.534566 15796 mvcc.cc:205] Tried to move back new op lower bound from 7316093393003335680 to 7316093392913096704. Current Snapshot: MvccSnapshot[applied={T|T < 7316093393003335680}]
I20260808 02:20:13.539831 15059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:13.539963 15059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:13.540743 15059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfd246ffd0d841c89cfde59c2137f311: Sending DeleteTablet for 3 replicas of tablet e91dc8f745f44c67a3e97edfc2711f43
I20260808 02:20:13.541424 15434 tablet_service.cc:1514] Processing DeleteTablet for tablet e91dc8f745f44c67a3e97edfc2711f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:13 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:45454
I20260808 02:20:13.541543 15228 tablet_service.cc:1514] Processing DeleteTablet for tablet e91dc8f745f44c67a3e97edfc2711f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:13 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:38180
I20260808 02:20:13.542107 15636 tablet_service.cc:1514] Processing DeleteTablet for tablet e91dc8f745f44c67a3e97edfc2711f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:13 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:51328
I20260808 02:20:13.542682 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 15107
I20260808 02:20:13.546396 15806 tablet_replica.cc:330] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: stopping tablet replica
I20260808 02:20:13.546586 15806 raft_consensus.cc:2229] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:20:13.546767 15806 raft_consensus.cc:2258] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:13.549531 15806 ts_tablet_manager.cc:1905] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:13.565096 15806 ts_tablet_manager.cc:1918] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:13.565264 15806 log.cc:1196] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/e91dc8f745f44c67a3e97edfc2711f43
I20260808 02:20:13.565847 15806 ts_tablet_manager.cc:1939] T e91dc8f745f44c67a3e97edfc2711f43 P 1fc2efd7aa904d8c8eb7bcb0015bdad1: Deleting consensus metadata
I20260808 02:20:13.568645 15040 catalog_manager.cc:4841] TS 1fc2efd7aa904d8c8eb7bcb0015bdad1 (127.28.177.130:35215): tablet e91dc8f745f44c67a3e97edfc2711f43 (table test-table [id=0f698e6b7376413dbd2a9d3a244161de]) successfully deleted
I20260808 02:20:13.575456 15808 tablet_replica.cc:330] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: stopping tablet replica
I20260808 02:20:13.575649 15808 raft_consensus.cc:2229] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:20:13.575802 15808 raft_consensus.cc:2258] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:13.576414 15808 ts_tablet_manager.cc:1905] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:13.581096 15808 ts_tablet_manager.cc:1918] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:13.581285 15808 log.cc:1196] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/e91dc8f745f44c67a3e97edfc2711f43
I20260808 02:20:13.581715 15808 ts_tablet_manager.cc:1939] T e91dc8f745f44c67a3e97edfc2711f43 P 657f47fe115d4234995247420fbb4c8f: Deleting consensus metadata
I20260808 02:20:13.583503 15043 catalog_manager.cc:4841] TS 657f47fe115d4234995247420fbb4c8f (127.28.177.131:44373): tablet e91dc8f745f44c67a3e97edfc2711f43 (table test-table [id=0f698e6b7376413dbd2a9d3a244161de]) successfully deleted
Aug 08 02:20:13 dist-test-slave-3h5h krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155612, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for HTTP/127.28.177.148@KRBTEST.COM
W20260808 02:20:13.626945 15043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:40057 (error 108)
I20260808 02:20:13.628322 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 15342
I20260808 02:20:13.635919 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:13.635741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40057 (local address 127.0.0.1:38196)
0808 02:20:13.635804 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:13.635872 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 02:20:13.636250 15043 catalog_manager.cc:4584] TS 4414b75d99c14874b4413cd36fa78782 (127.28.177.129:40057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e91dc8f745f44c67a3e97edfc2711f43: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40057: connect: Connection refused (error 111)
I20260808 02:20:13.642329 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 15533
I20260808 02:20:13.651301 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 15019
I20260808 02:20:13.665930 29382 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:20:14.021193 29382 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:20:14.021302 29382 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:20:14.376567 29382 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:20:14.376 UTC [13773] LOG: received smart shutdown request
2026-08-08 02:20:14.381 UTC [13773] LOG: background worker "logical replication launcher" (PID 13785) exited with exit code 1
2026-08-08 02:20:14.382 UTC [13779] LOG: shutting down
2026-08-08 02:20:14.561 UTC [13773] LOG: database system is shut down
2026-08-08T02:20:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55182 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:20:14 dist-test-slave-3h5h krb5kdc[15822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:20:14 dist-test-slave-3h5h krb5kdc[15822](info): set up 2 sockets
Aug 08 02:20:14 dist-test-slave-3h5h krb5kdc[15822](info): commencing operation
krb5kdc: starting...
W20260808 02:20:16.040194 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:20:16.180776 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:39241
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:39241,127.28.177.189:41123,127.28.177.188:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:16.266991 15840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:16.267207 15840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:16.267231 15840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:16.268303 15840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:16.268337 15840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:16.268358 15840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:16.268373 15840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:16.268384 15840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:16.269863 15840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:39241,127.28.177.189:41123,127.28.177.188:42661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:16.270068 15840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:16.270246 15840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:16.275626 15850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:16.275717 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:16.275720 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:16.276069 15840 server_base.cc:1034] running on GCE node
I20260808 02:20:16.276387 15840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:16.276695 15840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:16.277876 15840 hybrid_clock.cc:648] HybridClock initialized: now 1786155616277835 us; error 60 us; skew 500 ppm
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:16.280764 15840 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:16.281503 15840 webserver.cc:460] Webserver started at http://127.28.177.190:37213/ using document root <none> and password file <none>
I20260808 02:20:16.281769 15840 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:16.281850 15840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:16.282032 15840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:16.283313 15840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "05a1259bf4374ce3bb8ca903853fd9e7"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "03cfb946cfcb4d437b86824909457882"
server_key_iv: "74dc4d0bde00effb603274da290021c1"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.283758 15840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "05a1259bf4374ce3bb8ca903853fd9e7"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "03cfb946cfcb4d437b86824909457882"
server_key_iv: "74dc4d0bde00effb603274da290021c1"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.285539 15840 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 02:20:16.286478 15858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:16.286693 15840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:16.286816 15840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "05a1259bf4374ce3bb8ca903853fd9e7"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "03cfb946cfcb4d437b86824909457882"
server_key_iv: "74dc4d0bde00effb603274da290021c1"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.286899 15840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:16.308041 15840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:16.310995 15840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:16.311134 15840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:16.315500 15840 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:39241
I20260808 02:20:16.315588 15928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:39241 every 8 connection(s)
I20260808 02:20:16.316004 15840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:16.317152 15929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:16.317586 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 15840
I20260808 02:20:16.317672 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:20:16.317812 29382 external_mini_cluster.cc:1428] Setting key 29e5936ce5e1676951aca863236f52a8
I20260808 02:20:16.319052 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.177.189@KRBTEST.COM; defaulting to no policy
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.189@KRBTEST.COM" created.
I20260808 02:20:16.330291 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.319861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:39241 (local address 127.0.0.1:55102)
0808 02:20:16.320068 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.320115 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.320611 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.320960 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.320963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.321269 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.321429 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.321434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.322725 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.322726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.323477 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.323479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.324066 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.324067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.324140 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.324783 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.324794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.326563 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.328601 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.328608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.328613 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.328841 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.329167 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.329168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.329169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.329288 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.329810 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.329817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.329880 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.329961 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.330093 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260808 02:20:16.330595 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.319887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.28.177.190:39241)
0808 02:20:16.320116 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.320127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.320231 (+ 104us) server_negotiation.cc:407] Connection header received
0808 02:20:16.320790 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.320793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.320816 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.320954 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.321579 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.322605 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.323597 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.323949 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.324097 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.326661 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.326669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.326670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.326677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.328497 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.328984 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.328988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.328988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.329090 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.329363 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.329364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.329364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.329725 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.329826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.330022 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.330197 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
I20260808 02:20:16.331292 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:20:16.331712 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.331565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52782)
0808 02:20:16.331647 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.331693 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 02:20:16.331877 15859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
I20260808 02:20:16.332424 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.332334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52788)
0808 02:20:16.332383 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.332409 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 02:20:16.332564 15929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:41123: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab.
Entry for principal kudu/127.28.177.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab.
I20260808 02:20:16.349597 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:41123
--webserver_interface=127.28.177.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab
--principal=kudu/127.28.177.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:39241,127.28.177.189:41123,127.28.177.188:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:16.381848 15929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } attempt: 1
I20260808 02:20:16.382411 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.382252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52794)
0808 02:20:16.382333 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.382377 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 02:20:16.382795 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.382693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52804)
0808 02:20:16.382741 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.382769 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 02:20:16.382889 15929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:41123: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
W20260808 02:20:16.439366 15939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:16.439646 15939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:16.439689 15939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:16.440888 15939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:16.440972 15939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:16.440996 15939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:16.441018 15939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:16.441051 15939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:16.442587 15939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:39241,127.28.177.189:41123,127.28.177.188:42661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab
--principal=kudu/127.28.177.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:41123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.177.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:16.442879 15939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:16.443125 15939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:16.448412 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:16.448419 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:16.448453 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:16.448765 15939 server_base.cc:1034] running on GCE node
I20260808 02:20:16.449084 15939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:16.449385 15939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:16.450551 15939 hybrid_clock.cc:648] HybridClock initialized: now 1786155616450515 us; error 64 us; skew 500 ppm
I20260808 02:20:16.451133 15929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } attempt: 2
I20260808 02:20:16.451617 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.451436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52810)
0808 02:20:16.451536 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.451577 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:20:16.452165 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.451938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52824)
0808 02:20:16.452101 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.452138 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260808 02:20:16.452298 15929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:41123: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41123: connect: Connection refused (error 111)
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:16.453819 15939 init.cc:377] Logged in from keytab as kudu/127.28.177.189@KRBTEST.COM (short username kudu)
I20260808 02:20:16.454226 15939 webserver.cc:460] Webserver started at http://127.28.177.189:44265/ using document root <none> and password file <none>
I20260808 02:20:16.454416 15939 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:16.454459 15939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:16.454558 15939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:16.455623 15939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data/instance:
uuid: "b7ca4518febf4199ae1e6b3357b44f92"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "a19394c0c719a9d869d1fd2744794ca6"
server_key_iv: "8ddc3761279600c0f01d7e1ca4143a84"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.455971 15939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal/instance:
uuid: "b7ca4518febf4199ae1e6b3357b44f92"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "a19394c0c719a9d869d1fd2744794ca6"
server_key_iv: "8ddc3761279600c0f01d7e1ca4143a84"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.457442 15939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:20:16.458287 15960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:16.458520 15939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:16.458624 15939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal
uuid: "b7ca4518febf4199ae1e6b3357b44f92"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "a19394c0c719a9d869d1fd2744794ca6"
server_key_iv: "8ddc3761279600c0f01d7e1ca4143a84"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.458703 15939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:16.473469 15939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:16.476025 15939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:16.476151 15939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:16.480896 15939 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.189:41123
I20260808 02:20:16.480983 16032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.189:41123 every 8 connection(s)
I20260808 02:20:16.481375 15939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
I20260808 02:20:16.482496 16033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:16.484453 16033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:20:16.486501 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 15939
I20260808 02:20:16.486626 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-1/wal/instance
I20260808 02:20:16.486877 29382 external_mini_cluster.cc:1428] Setting key 8bb9beeaed3383f243fbd70d6e53668c
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.188@KRBTEST.COM; defaulting to no policy
I20260808 02:20:16.496804 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.485458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.28.177.190:39241)
0808 02:20:16.485541 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.485543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.485804 (+ 261us) server_negotiation.cc:407] Connection header received
0808 02:20:16.486390 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.486391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.486406 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.486507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.487554 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.488066 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.489552 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.489934 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.490034 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.493300 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.493301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.493302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.493309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.495034 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.495655 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.495656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.495656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.495717 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.496039 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.496040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.496040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.496211 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.496271 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.496658 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.496735 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:20:16.497105 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.485408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:39241 (local address 127.0.0.1:55114)
0808 02:20:16.485648 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.485699 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.486260 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.486598 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.486604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.487049 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.487382 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.487390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.488191 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.488192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.489445 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.489449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.490065 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.490067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.490211 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.490922 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.490934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.493135 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.495188 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.495197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.495200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.495558 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.495798 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.495799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.495800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.495958 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.496293 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.496302 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.496505 (+ 203us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.496639 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.496785 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":138,"threads_started":1}
I20260808 02:20:16.497855 16033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.188@KRBTEST.COM" created.
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:20:16.506614 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.498167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52830)
0808 02:20:16.498244 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.498255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.498296 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.498558 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.498559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.498720 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.498778 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.498779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.499901 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.499902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.500400 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.500401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.501239 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.501241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.501305 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.501812 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.501819 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.503363 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.505436 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.505437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.505437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.505611 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.505855 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.505855 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.505856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.505917 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.506230 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.506231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.506271 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.506345 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.506453 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:20:16.506728 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.498209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.28.177.189:41123)
0808 02:20:16.498393 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.498404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.498420 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:20:16.498439 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.498441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.498453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.498542 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.498883 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.499797 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.500483 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.501013 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.501090 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.503462 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.503478 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.503478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.503485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.505335 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.505712 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.505716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.505717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.505778 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.505998 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.505999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.505999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.506154 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.506255 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.506377 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.506537 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260808 02:20:16.507304 16033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:20:16.507701 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.507574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42648)
0808 02:20:16.507625 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.507677 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 02:20:16.507839 15961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
I20260808 02:20:16.508595 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.508435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42662)
0808 02:20:16.508535 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.508573 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260808 02:20:16.508721 16033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:42661: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab.
Entry for principal kudu/127.28.177.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab.
I20260808 02:20:16.520937 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:42661
--webserver_interface=127.28.177.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab
--principal=kudu/127.28.177.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:39241,127.28.177.189:41123,127.28.177.188:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:16.543459 15929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } attempt: 3
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:20:16.551163 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.543851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52842)
0808 02:20:16.543924 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.543937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.544007 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.544183 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.544185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.544357 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.544431 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.544433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.545327 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.545329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.545857 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.545859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.546258 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.546259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.546341 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.546867 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.546872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.548116 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.550006 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.550007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.550007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.550227 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.550455 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.550456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.550456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.550531 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.550804 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.550805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.550854 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.550950 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.551058 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:20:16.551163 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.543967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.28.177.189:41123)
0808 02:20:16.544053 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.544055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.544064 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:20:16.544098 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.544099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.544110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.544177 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.544534 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.545240 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.545957 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.546178 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.546261 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.548211 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.548212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.548213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.548222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.549913 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.550318 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.550320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.550320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.550382 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.550607 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.550608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.550608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.550725 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.550800 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.550977 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.551062 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:20:16.551617 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:20:16.552042 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.551928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42672)
0808 02:20:16.551986 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.552021 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:20:16.552491 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.552307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42684)
0808 02:20:16.552408 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.552468 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260808 02:20:16.552734 15929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:42661: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
I20260808 02:20:16.557945 16033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } attempt: 1
I20260808 02:20:16.558435 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.558298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42688)
0808 02:20:16.558363 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.558399 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:20:16.558864 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.558763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42704)
0808 02:20:16.558812 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.558839 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 02:20:16.558966 16033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:42661: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
I20260808 02:20:16.583941 15929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } attempt: 1
I20260808 02:20:16.584583 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.584426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42712)
0808 02:20:16.584499 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.584547 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:20:16.585029 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.584921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42728)
0808 02:20:16.584973 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.585001 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 02:20:16.585127 15929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:42661: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
W20260808 02:20:16.610335 16043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:16.610582 16043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:16.610608 16043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:16.611831 16043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:16.611883 16043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:16.611900 16043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:16.611920 16043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:16.611970 16043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:16.613655 16043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:39241,127.28.177.189:41123,127.28.177.188:42661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab
--principal=kudu/127.28.177.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.177.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:16.613982 16043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:16.614244 16043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:16.619607 16050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:16.619607 16051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:16.619645 16055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:16.619894 16043 server_base.cc:1034] running on GCE node
I20260808 02:20:16.620253 16043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:16.620566 16043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:16.621762 16043 hybrid_clock.cc:648] HybridClock initialized: now 1786155616621763 us; error 46 us; skew 500 ppm
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:16.624655 16043 init.cc:377] Logged in from keytab as kudu/127.28.177.188@KRBTEST.COM (short username kudu)
I20260808 02:20:16.625300 16043 webserver.cc:460] Webserver started at http://127.28.177.188:40479/ using document root <none> and password file <none>
I20260808 02:20:16.625573 16043 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:16.625650 16043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:16.625823 16043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:16.627117 16033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } attempt: 2
I20260808 02:20:16.627025 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data/instance:
uuid: "eb7a2f1c52e6410680575bfaccf77a09"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "43a925cb1381c204a0dd3c3fd99913b7"
server_key_iv: "bcd9ca762b550cbf0fe8c8d34fc65624"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.627480 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal/instance:
uuid: "eb7a2f1c52e6410680575bfaccf77a09"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "43a925cb1381c204a0dd3c3fd99913b7"
server_key_iv: "bcd9ca762b550cbf0fe8c8d34fc65624"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.627562 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.627404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42744)
0808 02:20:16.627502 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.627533 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 02:20:16.628016 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.627895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42758)
0808 02:20:16.627961 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.627990 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 02:20:16.628150 16033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:42661: Network error: Client connection negotiation failed: client connection to 127.28.177.188:42661: connect: Connection refused (error 111)
I20260808 02:20:16.629562 16043 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260808 02:20:16.630522 16065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:16.630766 16043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:20:16.630877 16043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal
uuid: "eb7a2f1c52e6410680575bfaccf77a09"
format_stamp: "Formatted at 2026-08-08 02:20:16 on dist-test-slave-3h5h"
server_key: "43a925cb1381c204a0dd3c3fd99913b7"
server_key_iv: "bcd9ca762b550cbf0fe8c8d34fc65624"
server_key_version: "encryptionkey@0"
I20260808 02:20:16.631081 16043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:16.641952 16043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:16.644425 16043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:16.644647 16043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:16.649029 16043 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.188:42661
I20260808 02:20:16.649097 16129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.188:42661 every 8 connection(s)
I20260808 02:20:16.649494 16043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
I20260808 02:20:16.650480 16131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:16.652323 16131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:20:16.657069 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 16043
I20260808 02:20:16.657219 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155444117209-29382-0/minicluster-data/master-2/wal/instance
I20260808 02:20:16.657456 29382 external_mini_cluster.cc:1428] Setting key 69830fe139abe82e8af71615f3b3399d
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:16.660274 15929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } attempt: 2
I20260808 02:20:16.663364 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.653473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.28.177.190:39241)
0808 02:20:16.653552 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.653555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.653754 (+ 199us) server_negotiation.cc:407] Connection header received
0808 02:20:16.654331 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.654332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.654346 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.654412 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.655077 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.655564 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.656581 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.656806 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.656931 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.659443 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.659444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.659445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.659453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.661411 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.661936 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.661937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.661938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.662008 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.662545 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.662546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.662547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.662757 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.662839 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.663176 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.663267 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":39}
I20260808 02:20:16.663580 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.653362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:39241 (local address 127.0.0.1:55118)
0808 02:20:16.653602 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.653652 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.654207 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.654493 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.654497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.654765 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.654974 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.654979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.655669 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.655671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.656472 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.656474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.656943 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.656945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.657082 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.657685 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.657696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.659340 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.661512 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.661522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.661524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.661853 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.662332 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.662333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.662334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.662463 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.662875 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.662883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.663017 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.663136 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.663307 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:20:16.664135 16131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:20:16.669283 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.657907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.28.177.188:42661)
0808 02:20:16.658208 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.658213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.658234 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:20:16.658256 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.658258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.658283 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.658376 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.659167 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.659953 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.661246 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.661591 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.661690 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.664795 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.664803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.664804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.664812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.667025 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.667894 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.667897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.667897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.667976 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.668219 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.668220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.668220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.668399 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.668540 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.669050 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.669160 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":218,"threads_started":1}
I20260808 02:20:16.671437 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.660560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42782)
0808 02:20:16.660617 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.660626 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.660657 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.660986 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.660988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.661528 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.661573 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.661576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.662585 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.662587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.663531 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.663535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.664108 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.664109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.664176 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.664541 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.664548 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.666831 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.669605 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.669606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.669606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.669842 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.670140 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.670141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.670141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.670213 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.671120 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.671122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.671173 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.671245 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.671344 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 02:20:16.671463 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.660634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.28.177.188:42661)
0808 02:20:16.660811 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.660813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.660821 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:20:16.660875 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.660876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.660888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.660963 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.661663 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.662468 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.663646 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.664006 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.664329 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.666932 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.666933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.666934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.666942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.669490 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.669927 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.669928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.669928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.669992 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.670880 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.670880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.670881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.671038 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.671109 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.671272 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.671344 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260808 02:20:16.671916 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.664634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41123 (local address 127.0.0.1:52846)
0808 02:20:16.664678 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.664689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.664725 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.664872 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.664873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.665015 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.665062 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.665085 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.665813 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.665814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.666458 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.666461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.666955 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.666956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.667036 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.667598 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.667603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.669153 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.670882 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.670883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.670883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.671125 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.671328 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.671329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.671330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.671387 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.671660 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.671660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.671699 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.671760 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.671857 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 02:20:16.672310 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.664691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.28.177.189:41123)
0808 02:20:16.664735 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.664736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.664747 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:20:16.664787 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.664787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.664796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.664868 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.665171 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.665699 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.666576 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.666884 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.666945 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.669249 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.669251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.669251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.669260 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.670799 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.671189 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.671191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.671191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.671252 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.671448 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.671448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.671449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.671580 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.671635 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.672126 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.672208 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:20:16.673278 16131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:20:16.674441 15929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: Bootstrap starting.
I20260808 02:20:16.675227 15929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: Neither blocks nor log segments found. Creating new log.
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:16.675594 15929 log.cc:824] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:16.676604 15929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: No bootstrap required, opened a new log
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:20:16.678998 15929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.679152 15929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:16.679243 15929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a1259bf4374ce3bb8ca903853fd9e7, State: Initialized, Role: FOLLOWER
I20260808 02:20:16.679399 15929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.679519 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.670338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.28.177.190:39241)
0808 02:20:16.670377 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.670379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.670386 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:20:16.670430 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.670430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.670440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.670501 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.671013 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.671709 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.672776 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.673119 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.673203 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.676006 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.676007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.676007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.676014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.678041 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.678442 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.678444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.678444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.678513 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.678777 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.678778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.678779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.678923 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.679044 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.679292 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.679372 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:20:16.679927 15929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:16.680053 16140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.680243 16140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:20:16.681217 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.673573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42794)
0808 02:20:16.673615 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.673624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.673664 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.673848 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.673849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.673998 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.674032 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.674034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.674977 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.674978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.675570 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.675573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.676008 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.676009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.676070 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.676455 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.676459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.677896 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.679967 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.679968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.679969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.680211 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.680449 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.680450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.680450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.680515 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.680883 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.680884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.680929 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.680983 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.681081 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 02:20:16.681703 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.673657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.28.177.188:42661)
0808 02:20:16.673712 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.673715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.673725 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:20:16.673748 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.673749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.673760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.673833 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.674129 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.674880 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.675655 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.675902 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.676221 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.678002 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.678003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.678004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.678011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.679857 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.680302 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.680303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.680304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.680361 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.680593 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.680593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.680594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.680774 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.681493 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.681535 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.681614 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 02:20:16.682346 16154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:20:16.682456 16154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:20:16.683768 16131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09: Bootstrap starting.
I20260808 02:20:16.684571 16131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:16.684975 16131 log.cc:824] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:16.685812 16131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09: No bootstrap required, opened a new log
I20260808 02:20:16.683324 15840 master_runner.cc:186] Error getting master registration for 127.28.177.189:41123: OK, instance_id { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" instance_seqno: 1786155616479518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:20:16.685913 15840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:20:16.687704 16131 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.687856 16131 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:16.687907 16131 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7a2f1c52e6410680575bfaccf77a09, State: Initialized, Role: FOLLOWER
I20260808 02:20:16.688040 16131 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.688405 16131 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:20:16.689121 16157 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.689285 16157 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:20:16.690553 16169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:20:16.690662 16169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:20:16.693537 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.680604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.28.177.189:41123)
0808 02:20:16.680648 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.680650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.680659 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:20:16.680686 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.680687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.680699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.680756 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.681467 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.682176 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.686130 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.686416 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.686505 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.689510 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.689512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.689512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.689522 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.691785 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.692226 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.692240 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.692240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.692307 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.692517 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.692518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.692519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.692682 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.692908 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.693361 (+ 453us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.693442 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:20:16.692899 16043 master_runner.cc:186] Error getting master registration for 127.28.177.189:41123: OK, instance_id { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" instance_seqno: 1786155616479518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:20:16.695017 16043 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:20:16.696251 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:20:16.705206 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.697802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.28.177.189:41123)
0808 02:20:16.697847 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.697849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.697972 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:20:16.698090 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.698091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.698103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.698134 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.698881 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.699370 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.700088 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.700296 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.700374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.702059 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.702061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.702062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.702070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.703803 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.704213 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.704214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.704215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.704276 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.704507 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.704509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.704509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.704651 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.704731 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.704993 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.705077 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:20:16.709249 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.698210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.28.177.190:39241)
0808 02:20:16.698252 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.698254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.701201 (+ 2947us) server_negotiation.cc:407] Connection header received
0808 02:20:16.701302 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.701303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.701317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.701351 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.702120 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.702852 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.703914 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.704227 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.704270 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.705823 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.705825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.706104 (+ 279us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.706115 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.707668 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.708248 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.708249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.708250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.708312 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.708566 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.708567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.708567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.708698 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.708769 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.709060 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.709150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:20:16.709249 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.698069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.28.177.188:42661)
0808 02:20:16.698114 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.698116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.701013 (+ 2897us) server_negotiation.cc:407] Connection header received
0808 02:20:16.701075 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.701076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.701090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.701165 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.701900 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.702597 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.703620 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.703897 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.703942 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.705597 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.705599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.705599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.705607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.707669 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.708248 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.708249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.708250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.708312 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.708567 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.708567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.708568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.708698 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.708782 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.709067 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.709150 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:20:16.719306 16033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } attempt: 3
Aug 08 02:20:16 dist-test-slave-3h5h krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155616, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:20:16.724871 16157 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:20:16.724992 16157 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.725678 16157 leader_election.cc:288] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05a1259bf4374ce3bb8ca903853fd9e7 (127.28.177.190:39241), b7ca4518febf4199ae1e6b3357b44f92 (127.28.177.189:41123)
I20260808 02:20:16.726070 15895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7a2f1c52e6410680575bfaccf77a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" is_pre_election: true
I20260808 02:20:16.726253 15895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb7a2f1c52e6410680575bfaccf77a09 in term 0.
I20260808 02:20:16.726905 16069 leader_election.cc:302] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a1259bf4374ce3bb8ca903853fd9e7, eb7a2f1c52e6410680575bfaccf77a09; no voters:
I20260808 02:20:16.727504 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.719684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:42661 (local address 127.0.0.1:42812)
0808 02:20:16.719761 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:16.719774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:16.719826 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:16.720185 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:16.720186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:16.720407 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:16.720455 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.720457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.721216 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.721218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.721908 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:16.721911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.722360 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.722361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:16.722425 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.722939 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:16.722944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:16.724236 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:16.726169 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.726170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.726171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.726424 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.726810 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:16.726811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:16.726811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:16.726886 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:16.727181 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:16.727182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:16.727229 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:20:16.727279 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:20:16.727392 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:20:16.728111 16157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:20:16.728205 16157 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:20:16.728237 16157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:16.729146 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:16.719732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.28.177.188:42661)
0808 02:20:16.719775 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:16.719777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:16.719809 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:20:16.719910 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:16.719912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:16.719923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:16.720132 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:16.720541 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.721130 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.722000 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:16.722276 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:16.722345 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:16.724326 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:16.724328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:16.724328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:16.724336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:16.726074 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.726643 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.726644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.726644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.726717 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:16.726961 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:16.726962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:16.726962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:16.727102 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:16.727940 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:16.727989 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 02:20:16.728079 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:20:16.729169 16157 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.729490 16157 leader_election.cc:288] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [CANDIDATE]: Term 1 election: Requested vote from peers 05a1259bf4374ce3bb8ca903853fd9e7 (127.28.177.190:39241), b7ca4518febf4199ae1e6b3357b44f92 (127.28.177.189:41123)
I20260808 02:20:16.731014 16033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: Bootstrap starting.
I20260808 02:20:16.731804 16033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:16.732141 16033 log.cc:824] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:16.732646 15895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7a2f1c52e6410680575bfaccf77a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a1259bf4374ce3bb8ca903853fd9e7"
I20260808 02:20:16.732764 15895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:16.733234 16033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: No bootstrap required, opened a new log
I20260808 02:20:16.733841 15895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7a2f1c52e6410680575bfaccf77a09 in term 1.
I20260808 02:20:16.734119 16069 leader_election.cc:302] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a1259bf4374ce3bb8ca903853fd9e7, eb7a2f1c52e6410680575bfaccf77a09; no voters:
I20260808 02:20:16.734265 16157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:16.734434 16157 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [term 1 LEADER]: Becoming Leader. State: Replica: eb7a2f1c52e6410680575bfaccf77a09, State: Running, Role: LEADER
I20260808 02:20:16.734534 16157 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.734953 16157 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7a2f1c52e6410680575bfaccf77a09. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.735073 16157 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:16.735637 16188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:16.735896 16188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:16.737777 16033 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.738065 16033 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:16.738227 16033 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ca4518febf4199ae1e6b3357b44f92, State: Initialized, Role: FOLLOWER
I20260808 02:20:16.738585 15895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [term 1 FOLLOWER]: Refusing update from remote peer eb7a2f1c52e6410680575bfaccf77a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:16.738499 16033 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } }
I20260808 02:20:16.739058 16157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:16.739151 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7a2f1c52e6410680575bfaccf77a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ca4518febf4199ae1e6b3357b44f92" is_pre_election: true
I20260808 02:20:16.739208 15994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:16.739425 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7a2f1c52e6410680575bfaccf77a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ca4518febf4199ae1e6b3357b44f92"
I20260808 02:20:16.739609 16033 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:16.740192 15994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [term 1 FOLLOWER]: Refusing update from remote peer eb7a2f1c52e6410680575bfaccf77a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:16.740968 16140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7a2f1c52e6410680575bfaccf77a09. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.741107 16140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:20:16.741319 16157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:16.744010 16157 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.744119 16157 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:16.744207 16189 mvcc.cc:205] Tried to move back new op lower bound from 7316093406157230080 to 7316093406145417216. Current Snapshot: MvccSnapshot[applied={T|T < 7316093406157230080}]
I20260808 02:20:16.745026 16188 catalog_manager.cc:1347] Generated new cluster ID: 2561137c8ecf43de8bf6bd15833a4f04
I20260808 02:20:16.745098 16188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:16.745396 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.745574 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:20:16.745663 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7a2f1c52e6410680575bfaccf77a09. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.745748 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:20:16.745814 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.745895 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:20:16.746871 16157 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.746968 16157 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09 [sys.catalog]: This master's current role is: LEADER
W20260808 02:20:16.747466 16209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 02:20:16.747475 16140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb7a2f1c52e6410680575bfaccf77a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a1259bf4374ce3bb8ca903853fd9e7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39241 } } peers { permanent_uuid: "b7ca4518febf4199ae1e6b3357b44f92" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41123 } } peers { permanent_uuid: "eb7a2f1c52e6410680575bfaccf77a09" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 42661 } } }
I20260808 02:20:16.747573 16140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:20:16.747661 16209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:20:16.752997 16198 mvcc.cc:205] Tried to move back new op lower bound from 7316093406157230080 to 7316093406145417216. Current Snapshot: MvccSnapshot[applied={T|T < 7316093406157230080}]
I20260808 02:20:16.759001 16188 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:20:16.760073 16188 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:16.772207 16188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7a2f1c52e6410680575bfaccf77a09: Generated new TSK 0
I20260808 02:20:16.772475 16188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:20:17.683066 16154 catalog_manager.cc:1259] Loaded cluster ID: 2561137c8ecf43de8bf6bd15833a4f04
I20260808 02:20:17.683159 16154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: loading cluster ID for follower catalog manager: success
I20260808 02:20:17.684003 16154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: acquiring CA information for follower catalog manager: success
I20260808 02:20:17.684259 16154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05a1259bf4374ce3bb8ca903853fd9e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:20:17.748307 16209 catalog_manager.cc:1259] Loaded cluster ID: 2561137c8ecf43de8bf6bd15833a4f04
I20260808 02:20:17.748390 16209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: loading cluster ID for follower catalog manager: success
I20260808 02:20:17.749358 16209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: acquiring CA information for follower catalog manager: success
I20260808 02:20:17.749656 16209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7ca4518febf4199ae1e6b3357b44f92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:20:17.828464 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 15840
I20260808 02:20:17.840931 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 15939
I20260808 02:20:17.854269 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 16043
2026-08-08T02:20:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3178 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:20:17 dist-test-slave-3h5h krb5kdc[16234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:20:17 dist-test-slave-3h5h krb5kdc[16234](info): set up 2 sockets
Aug 08 02:20:17 dist-test-slave-3h5h krb5kdc[16234](info): commencing operation
krb5kdc: starting...
W20260808 02:20:19.173087 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:20:19 dist-test-slave-3h5h krb5kdc[16234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:20:19.308187 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37209
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:19.401098 16256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:19.401314 16256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:19.401335 16256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:19.402464 16256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:19.402503 16256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:19.402516 16256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:19.402529 16256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:19.402541 16256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:19.404222 16256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:37209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:19.404434 16256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:19.404620 16256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:19.409808 16266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:19.409796 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:19.409796 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:19.409874 16256 server_base.cc:1034] running on GCE node
I20260808 02:20:19.410365 16256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:19.410701 16256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:19.411861 16256 hybrid_clock.cc:648] HybridClock initialized: now 1786155619411838 us; error 40 us; skew 500 ppm
Aug 08 02:20:19 dist-test-slave-3h5h krb5kdc[16234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155619, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:19.414791 16256 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:19.415337 16256 webserver.cc:460] Webserver started at http://127.28.177.190:38979/ using document root <none> and password file <none>
I20260808 02:20:19.415619 16256 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:19.415670 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:19.415825 16256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:19.417049 16256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "46db817a24964b88b92ca94d9a2a21e9"
format_stamp: "Formatted at 2026-08-08 02:20:19 on dist-test-slave-3h5h"
server_key: "29fe853af7c6df33bbeb663f4c5757c9"
server_key_iv: "a162dd8f206f063d08a1cc3ad8396c85"
server_key_version: "encryptionkey@0"
I20260808 02:20:19.417425 16256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "46db817a24964b88b92ca94d9a2a21e9"
format_stamp: "Formatted at 2026-08-08 02:20:19 on dist-test-slave-3h5h"
server_key: "29fe853af7c6df33bbeb663f4c5757c9"
server_key_iv: "a162dd8f206f063d08a1cc3ad8396c85"
server_key_version: "encryptionkey@0"
I20260808 02:20:19.418807 16256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:20:19.419531 16274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:19.419739 16256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:20:19.419852 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "46db817a24964b88b92ca94d9a2a21e9"
format_stamp: "Formatted at 2026-08-08 02:20:19 on dist-test-slave-3h5h"
server_key: "29fe853af7c6df33bbeb663f4c5757c9"
server_key_iv: "a162dd8f206f063d08a1cc3ad8396c85"
server_key_version: "encryptionkey@0"
I20260808 02:20:19.419929 16256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:19.446731 16256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:19.449285 16256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:19.449493 16256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:19.453948 16256 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:37209
I20260808 02:20:19.454002 16353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:37209 every 8 connection(s)
I20260808 02:20:19.454377 16256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:19.454972 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 16256
I20260808 02:20:19.455093 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:20:19.455309 29382 external_mini_cluster.cc:1428] Setting key 03d4af10ddecf51991c14c15667d7de3
I20260808 02:20:19.455408 16354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:19.459054 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9: Bootstrap starting.
I20260808 02:20:19.459703 16354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:19.460023 16354 log.cc:824] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:19.460762 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9: No bootstrap required, opened a new log
Aug 08 02:20:19 dist-test-slave-3h5h krb5kdc[16234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:19.462914 16354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46db817a24964b88b92ca94d9a2a21e9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37209 } }
I20260808 02:20:19.463075 16354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:19.463138 16354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46db817a24964b88b92ca94d9a2a21e9, State: Initialized, Role: FOLLOWER
I20260808 02:20:19.463338 16354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46db817a24964b88b92ca94d9a2a21e9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37209 } }
I20260808 02:20:19.463433 16354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:19.463469 16354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:19.463519 16354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:19.464201 16354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46db817a24964b88b92ca94d9a2a21e9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37209 } }
I20260808 02:20:19.464311 16354 leader_election.cc:302] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46db817a24964b88b92ca94d9a2a21e9; no voters:
I20260808 02:20:19.464504 16354 leader_election.cc:288] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:20:19.464545 16361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:19.464663 16361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [term 1 LEADER]: Becoming Leader. State: Replica: 46db817a24964b88b92ca94d9a2a21e9, State: Running, Role: LEADER
I20260808 02:20:19.464788 16354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:19.464811 16361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46db817a24964b88b92ca94d9a2a21e9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37209 } }
I20260808 02:20:19.465241 16361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46db817a24964b88b92ca94d9a2a21e9. Latest consensus state: current_term: 1 leader_uuid: "46db817a24964b88b92ca94d9a2a21e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46db817a24964b88b92ca94d9a2a21e9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37209 } } }
I20260808 02:20:19.465380 16361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:19.465248 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46db817a24964b88b92ca94d9a2a21e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46db817a24964b88b92ca94d9a2a21e9" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37209 } } }
I20260808 02:20:19.465719 16373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:19.465796 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:19.466416 16373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:19.467116 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:19.456012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.28.177.190:37209)
0808 02:20:19.456223 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:19.456233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:19.456272 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:20:19.456726 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:19.456736 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:19.456800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:19.457007 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:19.457726 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:19.459084 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:19.459950 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:19.460423 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:19.460591 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:19.463056 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:19.463063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:19.463063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:19.463079 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:19.465217 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:19.465678 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:19.465681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:19.465682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:19.465793 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:19.466063 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:19.466064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:19.466064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:19.466409 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:19.466592 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:19.466715 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:20:19.466874 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260808 02:20:19.468276 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:20:19.468573 16373 catalog_manager.cc:1347] Generated new cluster ID: 33156113da1d4ba084897316e4a31c83
I20260808 02:20:19.468626 16373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:19.483168 16373 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:20:19.483835 16373 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:19.489912 16373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46db817a24964b88b92ca94d9a2a21e9: Generated new TSK 0
I20260808 02:20:19.490129 16373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:20:19.523120 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 16256
2026-08-08T02:20:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1669 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:20:19 dist-test-slave-3h5h krb5kdc[16393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:20:19 dist-test-slave-3h5h krb5kdc[16393](info): set up 2 sockets
Aug 08 02:20:19 dist-test-slave-3h5h krb5kdc[16393](info): commencing operation
krb5kdc: starting...
W20260808 02:20:20.774286 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:20:20 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:20:20.912911 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37135
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:37135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:20.999619 16410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:20.999892 16410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:20.999933 16410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:21.001237 16410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:21.001325 16410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:21.001350 16410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:21.001371 16410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:21.001415 16410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:21.003083 16410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:37135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:21.003391 16410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:21.003643 16410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:21.009159 16417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.009131 16422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.009174 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.009135 16410 server_base.cc:1034] running on GCE node
I20260808 02:20:21.009728 16410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:21.010016 16410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:21.011193 16410 hybrid_clock.cc:648] HybridClock initialized: now 1786155621011172 us; error 32 us; skew 500 ppm
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:21.018945 16410 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:21.019562 16410 webserver.cc:460] Webserver started at http://127.28.177.190:33027/ using document root <none> and password file <none>
I20260808 02:20:21.019809 16410 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:21.019858 16410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:21.020020 16410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:21.021217 16410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "f90d689397ef4fafbe822da41ae70cdd"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "ad6cfcea4d432b1e9a5d60030f0da12f"
server_key_iv: "66142a7e1f2420fd63e7c6cb21dcefaa"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.021605 16410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "f90d689397ef4fafbe822da41ae70cdd"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "ad6cfcea4d432b1e9a5d60030f0da12f"
server_key_iv: "66142a7e1f2420fd63e7c6cb21dcefaa"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.023154 16410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:20:21.024019 16430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.024326 16410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.024420 16410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "f90d689397ef4fafbe822da41ae70cdd"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "ad6cfcea4d432b1e9a5d60030f0da12f"
server_key_iv: "66142a7e1f2420fd63e7c6cb21dcefaa"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.024477 16410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:21.036636 16410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:21.039229 16410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:21.039369 16410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:21.043952 16410 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:37135
I20260808 02:20:21.044025 16500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:37135 every 8 connection(s)
I20260808 02:20:21.044446 16410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:21.045652 16501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:21.048237 16501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap starting.
I20260808 02:20:21.049197 16501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:21.049525 16501 log.cc:824] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:21.049535 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 16410
I20260808 02:20:21.049695 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:20:21.049921 29382 external_mini_cluster.cc:1428] Setting key 8746d6c067690134b0774a2925278b05
I20260808 02:20:21.050477 16501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: No bootstrap required, opened a new log
I20260808 02:20:21.052801 16501 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.053042 16501 raft_consensus.cc:374] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:21.053097 16501 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90d689397ef4fafbe822da41ae70cdd, State: Initialized, Role: FOLLOWER
I20260808 02:20:21.053287 16501 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.053385 16501 raft_consensus.cc:388] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:21.053434 16501 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:21.053486 16501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:21.054123 16501 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.054260 16501 leader_election.cc:302] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90d689397ef4fafbe822da41ae70cdd; no voters:
I20260808 02:20:21.054494 16501 leader_election.cc:288] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:20:21.054523 16506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:21.054724 16506 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 LEADER]: Becoming Leader. State: Replica: f90d689397ef4fafbe822da41ae70cdd, State: Running, Role: LEADER
I20260808 02:20:21.054857 16501 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:21.054869 16506 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.055401 16507 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f90d689397ef4fafbe822da41ae70cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } } }
I20260808 02:20:21.055502 16507 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:21.055714 16508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90d689397ef4fafbe822da41ae70cdd. Latest consensus state: current_term: 1 leader_uuid: "f90d689397ef4fafbe822da41ae70cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } } }
I20260808 02:20:21.056129 16508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:21.056144 16516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:21.056960 16516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:21.061882 16516 catalog_manager.cc:1347] Generated new cluster ID: 1e3448a43bb044ab8ff14eb57adaf52b
I20260808 02:20:21.061942 16516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:21.063536 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.050628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.28.177.190:37135)
0808 02:20:21.050927 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.050938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.050973 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:20:21.051951 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.051962 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.052033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.052212 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:21.052988 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.054430 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.055450 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.056088 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.056200 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.059197 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:21.059201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:21.059202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:21.059217 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:21.061663 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.062253 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.062257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.062257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.062366 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.062597 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.062598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.062598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.062864 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:21.063042 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.063161 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.063309 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":158,"threads_started":1}
I20260808 02:20:21.076754 16516 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:20:21.077500 16516 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:21.080506 16516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Generated new TSK 0
I20260808 02:20:21.080710 16516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:20:21.117421 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37135
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:21.208462 16531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:21.208698 16531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:21.208726 16531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:21.210007 16531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:21.210060 16531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:21.210129 16531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:20:21.211712 16531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:21.212023 16531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:21.212302 16531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:21.218231 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.218200 16539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.218290 16531 server_base.cc:1034] running on GCE node
W20260808 02:20:21.218195 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.218649 16531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:21.218922 16531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:21.220170 16531 hybrid_clock.cc:648] HybridClock initialized: now 1786155621220147 us; error 28 us; skew 500 ppm
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:21.222831 16531 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:20:21.223415 16531 webserver.cc:460] Webserver started at http://127.28.177.129:38745/ using document root <none> and password file <none>
I20260808 02:20:21.223659 16531 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:21.223727 16531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:21.223874 16531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:21.225037 16531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "cd1850d533634d40a08bf5c1c1c097c5"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "5e5d5655a4e48d24108c2466d682565f"
server_key_iv: "23f5a652d02d3d33b8ddf7036b9c0840"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.225404 16531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "cd1850d533634d40a08bf5c1c1c097c5"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "5e5d5655a4e48d24108c2466d682565f"
server_key_iv: "23f5a652d02d3d33b8ddf7036b9c0840"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.226796 16531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.227768 16553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.228025 16531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.228173 16531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "cd1850d533634d40a08bf5c1c1c097c5"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "5e5d5655a4e48d24108c2466d682565f"
server_key_iv: "23f5a652d02d3d33b8ddf7036b9c0840"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.228248 16531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:21.239487 16531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:21.241966 16531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:21.242100 16531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:21.242383 16531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:21.242730 16531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:21.242796 16531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.242841 16531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:21.242868 16531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.249311 16531 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:40333
I20260808 02:20:21.249365 16712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:40333 every 8 connection(s)
I20260808 02:20:21.249727 16531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:20:21.253788 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 16531
I20260808 02:20:21.253907 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:20:21.254117 29382 external_mini_cluster.cc:1428] Setting key 74777c7f8ecea70e3aa60e4cfca87c75
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:21.261119 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.250504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:54339 (local address 127.28.177.190:37135)
0808 02:20:21.250551 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.250553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.250780 (+ 227us) server_negotiation.cc:407] Connection header received
0808 02:20:21.251352 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.251353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.251369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.251419 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:21.252454 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.253040 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.254452 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.254780 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.254878 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.257560 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:21.257562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:21.257562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:21.257572 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:21.259071 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.259765 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.259767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.259767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.259838 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.260161 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.260162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.260162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.260375 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:21.260465 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.260929 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.261047 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":18}
I20260808 02:20:21.261344 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.250470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:54339)
0808 02:20:21.250642 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.250681 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.251235 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.251503 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.251507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.252131 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:21.252342 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.252347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.253145 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.253147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.254347 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.254350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.254892 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.254893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.255057 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.255719 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:21.255731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:21.257457 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:21.259246 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:21.259254 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:21.259257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:21.259650 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:21.259923 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:21.259924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:21.259925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:21.260083 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:21.260465 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:21.260474 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:21.260673 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.260805 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.261030 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260808 02:20:21.262130 16713 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37135
I20260808 02:20:21.262272 16713 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:21.262488 16713 heartbeater.cc:511] Master 127.28.177.190:37135 requested a full tablet report, sending...
I20260808 02:20:21.263125 16451 ts_manager.cc:194] Registered new tserver with Master: cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333)
I20260808 02:20:21.263782 16451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:54339
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:20:21.286281 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37135
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:21.374939 16718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:21.375216 16718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:21.375262 16718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:21.376454 16718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:21.376526 16718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:21.376570 16718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:20:21.378101 16718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:21.378382 16718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:21.378639 16718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:21.384373 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.384389 16724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.384479 16726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.384934 16718 server_base.cc:1034] running on GCE node
I20260808 02:20:21.385315 16718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:21.385623 16718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:21.386835 16718 hybrid_clock.cc:648] HybridClock initialized: now 1786155621386825 us; error 45 us; skew 500 ppm
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:21.390408 16718 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:20:21.390990 16718 webserver.cc:460] Webserver started at http://127.28.177.130:39629/ using document root <none> and password file <none>
I20260808 02:20:21.391289 16718 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:21.391371 16718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:21.391546 16718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:21.392738 16718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "c64eb94be4ce4a1fa7df039741c78316"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "acf0233c3ad2d8f3a65a1cb1ffb8f48e"
server_key_iv: "c47e2748a1bea9a8ba3d2b37540e8da2"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.393225 16718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "c64eb94be4ce4a1fa7df039741c78316"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "acf0233c3ad2d8f3a65a1cb1ffb8f48e"
server_key_iv: "c47e2748a1bea9a8ba3d2b37540e8da2"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.394850 16718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.395833 16738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.396063 16718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.396199 16718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "c64eb94be4ce4a1fa7df039741c78316"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "acf0233c3ad2d8f3a65a1cb1ffb8f48e"
server_key_iv: "c47e2748a1bea9a8ba3d2b37540e8da2"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.396301 16718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:21.410879 16718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:21.413242 16718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:21.413374 16718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:21.413610 16718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:21.413985 16718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:21.414047 16718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.414165 16718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:21.414206 16718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.420573 16718 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:37933
I20260808 02:20:21.420665 16894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:37933 every 8 connection(s)
I20260808 02:20:21.421048 16718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:20:21.422509 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 16718
I20260808 02:20:21.422607 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:20:21.422819 29382 external_mini_cluster.cc:1428] Setting key 86da091610f8f2d98c70369bd592dea4
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:21.432647 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.421921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:42535 (local address 127.28.177.190:37135)
0808 02:20:21.421973 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.421975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.422301 (+ 326us) server_negotiation.cc:407] Connection header received
0808 02:20:21.422940 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.422942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.422959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.423017 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:21.424309 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.424926 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.426511 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.426761 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.426861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.429483 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:21.429485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:21.429485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:21.429494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:21.431138 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.431559 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.431560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.431561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.431637 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.431915 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.431916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.431917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.432054 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:21.432140 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.432449 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.432539 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:20:21.432826 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.421857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:42535)
0808 02:20:21.422149 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.422198 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.422812 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.423099 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.423104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.423965 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:21.424189 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.424195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.425031 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.425032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.426405 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.426409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.426876 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.426878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.427062 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.427666 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:21.427675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:21.429365 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:21.431232 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:21.431240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:21.431243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:21.431476 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:21.431723 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:21.431724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:21.431725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:21.431842 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:21.432154 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:21.432162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:21.432304 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.432412 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.432584 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
I20260808 02:20:21.433531 16895 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37135
I20260808 02:20:21.433697 16895 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:21.433924 16895 heartbeater.cc:511] Master 127.28.177.190:37135 requested a full tablet report, sending...
I20260808 02:20:21.434393 16451 ts_manager.cc:194] Registered new tserver with Master: c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933)
I20260808 02:20:21.434795 16451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:42535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:20:21.457145 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37135
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:21.546571 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:21.546798 16901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:21.546823 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:21.547935 16901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:21.547969 16901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:21.547995 16901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:20:21.549489 16901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:21.549703 16901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:21.549896 16901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:21.555686 16909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.555764 16908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.555692 16911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.556100 16901 server_base.cc:1034] running on GCE node
I20260808 02:20:21.556421 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:21.556794 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:21.558063 16901 hybrid_clock.cc:648] HybridClock initialized: now 1786155621558049 us; error 28 us; skew 500 ppm
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:21.560920 16901 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:20:21.561506 16901 webserver.cc:460] Webserver started at http://127.28.177.131:40807/ using document root <none> and password file <none>
I20260808 02:20:21.561753 16901 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:21.561798 16901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:21.561972 16901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:21.563091 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "d6eeae4d6d9c4387bff813269619cade"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "1fc0e50b291ff3f846f68912a9b07ca2"
server_key_iv: "014282258711cf27461ec85272ae1662"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.563499 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "d6eeae4d6d9c4387bff813269619cade"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "1fc0e50b291ff3f846f68912a9b07ca2"
server_key_iv: "014282258711cf27461ec85272ae1662"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.565094 16901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:20:21.566002 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.566278 16901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.566412 16901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "d6eeae4d6d9c4387bff813269619cade"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "1fc0e50b291ff3f846f68912a9b07ca2"
server_key_iv: "014282258711cf27461ec85272ae1662"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.566500 16901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:21.577939 16901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:21.580084 16901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:21.580303 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:21.580631 16901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:21.581039 16901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:21.581099 16901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.581131 16901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:21.581149 16901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.588101 16901 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:40079
I20260808 02:20:21.588142 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:40079 every 8 connection(s)
I20260808 02:20:21.588526 16901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:20:21.594137 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 16901
I20260808 02:20:21.594238 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:20:21.594435 29382 external_mini_cluster.cc:1428] Setting key 35eacf210335d9d26cdca338839a5688
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:21.600291 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.589584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:43739 (local address 127.28.177.190:37135)
0808 02:20:21.589659 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.589661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.589837 (+ 176us) server_negotiation.cc:407] Connection header received
0808 02:20:21.590316 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.590318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.590331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.590399 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:21.591548 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.592141 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.593215 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.593416 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.593490 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.596583 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:21.596585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:21.596586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:21.596597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:21.598447 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.599036 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.599037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.599038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.599107 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.599377 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.599378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.599378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.599516 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:21.599651 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.600125 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.600203 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:20:21.600653 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.589440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:43739)
0808 02:20:21.589629 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.589710 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.590214 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.590485 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.590489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.591136 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:21.591438 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.591444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.592236 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.592237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.593131 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.593134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.593504 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.593504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.593607 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.594283 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:21.594296 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:21.596427 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:21.598564 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:21.598572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:21.598574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:21.598849 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:21.599196 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:21.599197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:21.599197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:21.599307 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:21.599693 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:21.599701 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:21.599855 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.600086 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.600249 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260808 02:20:21.601287 17081 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37135
I20260808 02:20:21.601389 17081 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:21.601544 17081 heartbeater.cc:511] Master 127.28.177.190:37135 requested a full tablet report, sending...
I20260808 02:20:21.601995 16451 ts_manager.cc:194] Registered new tserver with Master: d6eeae4d6d9c4387bff813269619cade (127.28.177.131:40079)
I20260808 02:20:21.602353 16451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:43739
I20260808 02:20:21.606681 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:20:21.614696 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.608118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.28.177.190:37135)
0808 02:20:21.608220 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.608222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.608311 (+ 89us) server_negotiation.cc:407] Connection header received
0808 02:20:21.608407 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.608408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.608420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.608493 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:21.609170 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.609669 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.610409 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.610612 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.610694 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.612004 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:21.612005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:21.612006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:21.612015 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:21.613434 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.613831 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.613833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.613833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.613896 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:21.614112 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:21.614113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:21.614113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:21.614284 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:21.614399 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.614501 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.614586 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:20:21.616708 16451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:20:21.618491 16451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:20:21.625910 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.621312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.28.177.130:37933)
0808 02:20:21.621957 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.621961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.621971 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:20:21.622095 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.622101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.622122 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.622348 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:21.622423 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.623447 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.624607 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.625383 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.625473 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.625534 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.625738 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.625783 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":107,"threads_started":1}
I20260808 02:20:21.625895 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.621081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:40079 (local address 127.0.0.1:49342)
0808 02:20:21.621498 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.621507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.621546 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.621968 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.621972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.621974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:21.622067 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.622070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.623500 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.623501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.624624 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.624626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.625508 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.625509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.625614 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.625623 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.625711 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.625790 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":59,"thread_start_us":90,"threads_started":1}
I20260808 02:20:21.625895 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.621068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37933 (local address 127.0.0.1:57688)
0808 02:20:21.621281 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.621327 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.621396 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.622224 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.622225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.622227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:21.622277 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.622279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.623535 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.623536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.624504 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.624506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.625507 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.625508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.625606 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.625623 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.625711 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.625790 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260808 02:20:21.626617 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.621471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.28.177.131:40079)
0808 02:20:21.621740 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.621746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.621757 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:20:21.621802 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.621805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.621845 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.622157 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:21.622239 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.623381 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.624712 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.625402 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.625474 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.625534 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.626482 (+ 948us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.626531 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":183,"threads_started":1}
I20260808 02:20:21.626984 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.621287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40333 (local address 127.0.0.1:55692)
0808 02:20:21.621756 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.621767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.621827 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.622264 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.622265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.622268 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:21.622318 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.622320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.623262 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.623263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.624619 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.624622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.626730 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.626732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.626794 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.626799 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.626856 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.626898 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":101,"threads_started":1}
I20260808 02:20:21.627306 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.621434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.28.177.129:40333)
0808 02:20:21.621660 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.621665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.621854 (+ 189us) server_negotiation.cc:407] Connection header received
0808 02:20:21.621907 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.621909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.621927 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.622163 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:21.622407 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.623150 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.625585 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.626344 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.626428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.626486 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.627173 (+ 687us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.627230 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":142,"threads_started":1}
I20260808 02:20:21.627203 16802 tablet_service.cc:1467] Processing CreateTablet for tablet 609bbcf780114dfa9be2335f3e44002f (DEFAULT_TABLE table=test-table [id=ec56ddccb54d47c0932248ddd3b20495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:21.627336 16979 tablet_service.cc:1467] Processing CreateTablet for tablet 609bbcf780114dfa9be2335f3e44002f (DEFAULT_TABLE table=test-table [id=ec56ddccb54d47c0932248ddd3b20495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:21.627635 16802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609bbcf780114dfa9be2335f3e44002f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:21.627720 16979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609bbcf780114dfa9be2335f3e44002f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:21.627957 16623 tablet_service.cc:1467] Processing CreateTablet for tablet 609bbcf780114dfa9be2335f3e44002f (DEFAULT_TABLE table=test-table [id=ec56ddccb54d47c0932248ddd3b20495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:21.628299 16623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609bbcf780114dfa9be2335f3e44002f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:21.630426 17102 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Bootstrap starting.
I20260808 02:20:21.631170 17102 tablet_bootstrap.cc:654] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:21.631188 17103 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Bootstrap starting.
I20260808 02:20:21.631899 17102 log.cc:824] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:21.631940 17103 tablet_bootstrap.cc:654] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:21.632047 17104 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Bootstrap starting.
I20260808 02:20:21.632591 17103 log.cc:824] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:21.632751 17104 tablet_bootstrap.cc:654] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:21.633014 17102 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: No bootstrap required, opened a new log
I20260808 02:20:21.633102 17102 ts_tablet_manager.cc:1397] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:20:21.633373 17103 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: No bootstrap required, opened a new log
I20260808 02:20:21.633378 17104 log.cc:824] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:21.633460 17103 ts_tablet_manager.cc:1397] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:20:21.634219 17104 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: No bootstrap required, opened a new log
I20260808 02:20:21.634305 17104 ts_tablet_manager.cc:1397] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:20:21.634986 17103 raft_consensus.cc:348] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.635133 17103 raft_consensus.cc:374] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:21.635156 17103 raft_consensus.cc:729] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64eb94be4ce4a1fa7df039741c78316, State: Initialized, Role: FOLLOWER
I20260808 02:20:21.635306 17103 consensus_queue.cc:260] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.635285 17102 raft_consensus.cc:348] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.635437 17102 raft_consensus.cc:374] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:21.635469 17102 raft_consensus.cc:729] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eeae4d6d9c4387bff813269619cade, State: Initialized, Role: FOLLOWER
I20260808 02:20:21.635584 17102 consensus_queue.cc:260] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.635628 17103 ts_tablet_manager.cc:1428] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:20:21.635696 16895 heartbeater.cc:503] Master 127.28.177.190:37135 was elected leader, sending a full tablet report...
I20260808 02:20:21.635857 17102 ts_tablet_manager.cc:1428] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:20:21.636053 17081 heartbeater.cc:503] Master 127.28.177.190:37135 was elected leader, sending a full tablet report...
I20260808 02:20:21.636472 17104 raft_consensus.cc:348] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.636657 17104 raft_consensus.cc:374] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:21.636724 17104 raft_consensus.cc:729] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1850d533634d40a08bf5c1c1c097c5, State: Initialized, Role: FOLLOWER
I20260808 02:20:21.636914 17104 consensus_queue.cc:260] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.637205 17104 ts_tablet_manager.cc:1428] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:20:21.637295 16713 heartbeater.cc:503] Master 127.28.177.190:37135 was elected leader, sending a full tablet report...
W20260808 02:20:21.671686 16896 tablet.cc:2369] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:20:21.681284 17110 raft_consensus.cc:482] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:20:21.681437 17110 raft_consensus.cc:504] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.681865 17110 leader_election.cc:288] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933), cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333)
I20260808 02:20:21.686507 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.682305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:39589 (local address 127.28.177.130:37933)
0808 02:20:21.682384 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.682386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.682398 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:20:21.682476 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.682478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.682489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.682548 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:21.682757 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.683717 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.685358 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.686025 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.686114 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.686123 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.686385 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.686434 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:20:21.686542 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.682444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:47281 (local address 127.28.177.129:40333)
0808 02:20:21.682530 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.682532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.682712 (+ 180us) server_negotiation.cc:407] Connection header received
0808 02:20:21.682784 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.682786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.682797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.682856 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:21.683144 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.683896 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.685387 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.686025 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.686114 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.686123 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.686385 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.686434 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:20:21.686560 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.682189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37933 (local address 127.28.177.131:39589)
0808 02:20:21.682297 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.682314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.682388 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.682550 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.682551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.682553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:21.682640 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.682642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.683794 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.683795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.685214 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.685225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.686145 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.686147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.686260 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.686265 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.686346 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.686396 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":38}
I20260808 02:20:21.686592 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.682348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40333 (local address 127.28.177.131:47281)
0808 02:20:21.682618 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.682629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:21.682719 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:21.682878 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:21.682879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:21.682880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:21.683040 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.683042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.683980 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.683980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.685286 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:21.685288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.686145 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.686147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:21.686263 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.686266 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:20:21.686346 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:20:21.686396 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":3,"thread_start_us":184,"threads_started":1}
I20260808 02:20:21.687016 16831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bbcf780114dfa9be2335f3e44002f" candidate_uuid: "d6eeae4d6d9c4387bff813269619cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64eb94be4ce4a1fa7df039741c78316" is_pre_election: true
I20260808 02:20:21.687156 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bbcf780114dfa9be2335f3e44002f" candidate_uuid: "d6eeae4d6d9c4387bff813269619cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1850d533634d40a08bf5c1c1c097c5" is_pre_election: true
I20260808 02:20:21.687242 16831 raft_consensus.cc:2454] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eeae4d6d9c4387bff813269619cade in term 0.
I20260808 02:20:21.687328 16649 raft_consensus.cc:2454] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eeae4d6d9c4387bff813269619cade in term 0.
I20260808 02:20:21.687595 16927 leader_election.cc:302] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64eb94be4ce4a1fa7df039741c78316, d6eeae4d6d9c4387bff813269619cade; no voters:
I20260808 02:20:21.687809 17110 raft_consensus.cc:2781] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:20:21.687896 17110 raft_consensus.cc:482] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:20:21.687915 17110 raft_consensus.cc:3037] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:21.688882 17110 raft_consensus.cc:504] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.689049 17110 leader_election.cc:288] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [CANDIDATE]: Term 1 election: Requested vote from peers c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933), cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333)
I20260808 02:20:21.689324 16831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bbcf780114dfa9be2335f3e44002f" candidate_uuid: "d6eeae4d6d9c4387bff813269619cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64eb94be4ce4a1fa7df039741c78316"
I20260808 02:20:21.689419 16831 raft_consensus.cc:3037] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:21.690295 16831 raft_consensus.cc:2454] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eeae4d6d9c4387bff813269619cade in term 1.
I20260808 02:20:21.690477 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bbcf780114dfa9be2335f3e44002f" candidate_uuid: "d6eeae4d6d9c4387bff813269619cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1850d533634d40a08bf5c1c1c097c5"
I20260808 02:20:21.690528 16927 leader_election.cc:302] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64eb94be4ce4a1fa7df039741c78316, d6eeae4d6d9c4387bff813269619cade; no voters:
I20260808 02:20:21.690614 17110 raft_consensus.cc:2781] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:21.690589 16649 raft_consensus.cc:3037] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:21.690801 17110 raft_consensus.cc:686] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 1 LEADER]: Becoming Leader. State: Replica: d6eeae4d6d9c4387bff813269619cade, State: Running, Role: LEADER
I20260808 02:20:21.690923 17110 consensus_queue.cc:237] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:21.691664 16649 raft_consensus.cc:2454] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eeae4d6d9c4387bff813269619cade in term 1.
I20260808 02:20:21.692077 16451 catalog_manager.cc:5495] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade reported cstate change: term changed from 0 to 1, leader changed from <none> to d6eeae4d6d9c4387bff813269619cade (127.28.177.131). New cstate: current_term: 1 leader_uuid: "d6eeae4d6d9c4387bff813269619cade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } health_report { overall_health: UNKNOWN } } }
I20260808 02:20:21.703588 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.701077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.28.177.131:40079)
0808 02:20:21.701144 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:21.701146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:21.701154 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:20:21.701199 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:21.701200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:21.701218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:21.701297 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:20:21.701426 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.701908 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.702738 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:21.702956 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:21.703036 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:21.703216 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:20:21.703330 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:20:21.703428 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:21.703453 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 02:20:21.703472 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:20:21.705754 16831 raft_consensus.cc:1261] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Refusing update from remote peer d6eeae4d6d9c4387bff813269619cade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:21.705753 16649 raft_consensus.cc:1261] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Refusing update from remote peer d6eeae4d6d9c4387bff813269619cade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:21.706126 17110 consensus_queue.cc:1035] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:21.706364 17113 consensus_queue.cc:1035] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:21.709544 17118 mvcc.cc:205] Tried to move back new op lower bound from 7316093426503290880 to 7316093426447130624. Current Snapshot: MvccSnapshot[applied={T|T < 7316093426503290880}]
I20260808 02:20:21.711501 16449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:21.711620 16449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:21.712448 16449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Sending DeleteTablet for 3 replicas of tablet 609bbcf780114dfa9be2335f3e44002f
I20260808 02:20:21.712785 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 16531
I20260808 02:20:21.712869 16623 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bbcf780114dfa9be2335f3e44002f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:21 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:55692
I20260808 02:20:21.715219 16979 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bbcf780114dfa9be2335f3e44002f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:21 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:49342
I20260808 02:20:21.722419 17133 tablet_replica.cc:330] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: stopping tablet replica
I20260808 02:20:21.722589 17133 raft_consensus.cc:2229] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:20:21.722759 17133 raft_consensus.cc:2258] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:21.723264 16802 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bbcf780114dfa9be2335f3e44002f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:21 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:57688
I20260808 02:20:21.724066 17134 tablet_replica.cc:330] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: stopping tablet replica
I20260808 02:20:21.724267 17134 raft_consensus.cc:2229] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:20:21.724478 17134 raft_consensus.cc:2258] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:21.725965 17134 ts_tablet_manager.cc:1905] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:21.729059 17134 ts_tablet_manager.cc:1918] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:21.729216 17134 log.cc:1196] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/609bbcf780114dfa9be2335f3e44002f
I20260808 02:20:21.729682 17134 ts_tablet_manager.cc:1939] T 609bbcf780114dfa9be2335f3e44002f P c64eb94be4ce4a1fa7df039741c78316: Deleting consensus metadata
I20260808 02:20:21.730424 16435 catalog_manager.cc:4841] TS c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933): tablet 609bbcf780114dfa9be2335f3e44002f (table test-table [id=ec56ddccb54d47c0932248ddd3b20495]) successfully deleted
I20260808 02:20:21.734848 17133 ts_tablet_manager.cc:1905] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:21.736733 17133 ts_tablet_manager.cc:1918] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:21.736848 17133 log.cc:1196] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/609bbcf780114dfa9be2335f3e44002f
I20260808 02:20:21.737385 17133 ts_tablet_manager.cc:1939] T 609bbcf780114dfa9be2335f3e44002f P d6eeae4d6d9c4387bff813269619cade: Deleting consensus metadata
I20260808 02:20:21.737946 16432 catalog_manager.cc:4841] TS d6eeae4d6d9c4387bff813269619cade (127.28.177.131:40079): tablet 609bbcf780114dfa9be2335f3e44002f (table test-table [id=ec56ddccb54d47c0932248ddd3b20495]) successfully deleted
W20260808 02:20:21.739625 16435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:40333 (error 108)
I20260808 02:20:21.740285 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:21.740136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40333 (local address 127.0.0.1:55704)
0808 02:20:21.740201 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:21.740248 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 02:20:21.740357 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 16718
W20260808 02:20:21.745085 16435 catalog_manager.cc:4584] TS cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 609bbcf780114dfa9be2335f3e44002f: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40333: connect: Connection refused (error 111)
I20260808 02:20:21.750921 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 16901
I20260808 02:20:21.763262 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 16410
I20260808 02:20:21.778803 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37135
--webserver_interface=127.28.177.190
--webserver_port=33027
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:37135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:21.864090 17136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:21.864326 17136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:21.864352 17136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:21.865638 17136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:21.865685 17136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:21.865700 17136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:21.865712 17136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:21.865725 17136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:21.867323 17136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:37135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:21.867616 17136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:21.867890 17136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:21.877911 17144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:21.877875 17147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.878260 17136 server_base.cc:1034] running on GCE node
W20260808 02:20:21.877892 17145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:21.878607 17136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:21.878945 17136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:21.880122 17136 hybrid_clock.cc:648] HybridClock initialized: now 1786155621880097 us; error 42 us; skew 500 ppm
Aug 08 02:20:21 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:21.883232 17136 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:21.883769 17136 webserver.cc:460] Webserver started at http://127.28.177.190:33027/ using document root <none> and password file <none>
I20260808 02:20:21.884083 17136 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:21.884182 17136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:21.885888 17136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.886727 17156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:21.887050 17136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:21.887189 17136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "f90d689397ef4fafbe822da41ae70cdd"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "ad6cfcea4d432b1e9a5d60030f0da12f"
server_key_iv: "66142a7e1f2420fd63e7c6cb21dcefaa"
server_key_version: "encryptionkey@0"
I20260808 02:20:21.887658 17136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:21.902104 17136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:21.904621 17136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:21.904803 17136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:21.909497 17136 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:37135
I20260808 02:20:21.909549 17231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:37135 every 8 connection(s)
I20260808 02:20:21.910001 17136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:21.911859 17232 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:20:21.912497 17232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap starting.
I20260808 02:20:21.914856 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 17136
I20260808 02:20:21.915215 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:40333
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:37135
--webserver_port=38745
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:21.922753 17232 log.cc:824] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:21.926326 17232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:20:21.926779 17232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap complete.
I20260808 02:20:21.930742 17232 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.931113 17232 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90d689397ef4fafbe822da41ae70cdd, State: Initialized, Role: FOLLOWER
I20260808 02:20:21.931339 17232 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.931440 17232 raft_consensus.cc:388] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:21.931526 17232 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:21.931617 17232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:20:21.933202 17232 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.933357 17232 leader_election.cc:302] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90d689397ef4fafbe822da41ae70cdd; no voters:
I20260808 02:20:21.933933 17232 leader_election.cc:288] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:20:21.933990 17237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:20:21.934198 17237 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 LEADER]: Becoming Leader. State: Replica: f90d689397ef4fafbe822da41ae70cdd, State: Running, Role: LEADER
I20260808 02:20:21.934252 17232 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:21.934330 17237 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:21.934597 17238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f90d689397ef4fafbe822da41ae70cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } } }
I20260808 02:20:21.934692 17238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:21.934595 17239 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90d689397ef4fafbe822da41ae70cdd. Latest consensus state: current_term: 2 leader_uuid: "f90d689397ef4fafbe822da41ae70cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } } }
I20260808 02:20:21.935006 17239 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:21.936295 17257 catalog_manager.cc:1259] Loaded cluster ID: 1e3448a43bb044ab8ff14eb57adaf52b
I20260808 02:20:21.936384 17257 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: loading cluster ID for follower catalog manager: success
W20260808 02:20:21.936863 17257 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260808 02:20:21.937001 17257 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260808 02:20:21.937110 17242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:21.937557 17242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:21.937690 17242 catalog_manager.cc:1259] Loaded cluster ID: 1e3448a43bb044ab8ff14eb57adaf52b
I20260808 02:20:21.937772 17242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:21.938002 17242 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 02:20:21.943018 17242 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 02:20:21.951627 17242 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:21.951998 17242 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Loaded TSK: 0
I20260808 02:20:21.952334 17242 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:20:22.029938 17235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:22.030174 17235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:22.030201 17235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:22.031472 17235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:22.031527 17235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:22.031601 17235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:20:22.033371 17235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:40333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=38745
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:22.033695 17235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:22.033982 17235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:22.040652 17267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.040657 17273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.041136 17235 server_base.cc:1034] running on GCE node
W20260808 02:20:22.040815 17268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.041507 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:22.041831 17235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:22.043002 17235 hybrid_clock.cc:648] HybridClock initialized: now 1786155622042972 us; error 45 us; skew 500 ppm
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:22.045971 17235 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:20:22.046558 17235 webserver.cc:460] Webserver started at http://127.28.177.129:38745/ using document root <none> and password file <none>
I20260808 02:20:22.046833 17235 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:22.046921 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:22.048424 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:20:22.049314 17282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.049515 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:20:22.049633 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "cd1850d533634d40a08bf5c1c1c097c5"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "5e5d5655a4e48d24108c2466d682565f"
server_key_iv: "23f5a652d02d3d33b8ddf7036b9c0840"
server_key_version: "encryptionkey@0"
I20260808 02:20:22.050011 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:22.080689 17235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:22.083324 17235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:22.083523 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:22.083829 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:22.084441 17289 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:20:22.086194 17235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:20:22.086254 17235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 02:20:22.086295 17235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:20:22.087077 17235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:20:22.087128 17235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 02:20:22.087170 17289 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Bootstrap starting.
I20260808 02:20:22.095924 17235 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:40333
I20260808 02:20:22.096037 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:40333 every 8 connection(s)
I20260808 02:20:22.096647 17235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:20:22.100379 17289 log.cc:824] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:22.104051 17289 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:20:22.104153 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 17235
I20260808 02:20:22.104494 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:37933
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:37135
--webserver_port=39629
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:22.104805 17289 tablet_bootstrap.cc:492] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Bootstrap complete.
I20260808 02:20:22.106451 17289 ts_tablet_manager.cc:1397] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260808 02:20:22.108438 17289 raft_consensus.cc:348] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.109032 17289 raft_consensus.cc:729] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1850d533634d40a08bf5c1c1c097c5, State: Initialized, Role: FOLLOWER
I20260808 02:20:22.109687 17289 consensus_queue.cc:260] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.110123 17289 ts_tablet_manager.cc:1428] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:22.117102 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.097661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:43677 (local address 127.28.177.190:37135)
0808 02:20:22.100858 (+ 3197us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.100872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.101294 (+ 422us) server_negotiation.cc:407] Connection header received
0808 02:20:22.103143 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.103149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.103200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.103365 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:22.104203 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.105787 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.107844 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.108648 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.108793 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.112111 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:22.112117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:22.112117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:22.112133 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:22.114669 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.115316 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.115319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.115320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.115436 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.115789 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.115789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.115790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.116178 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:22.116254 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.116707 (+ 453us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.116910 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3099,"thread_start_us":110,"threads_started":1}
I20260808 02:20:22.117360 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.097546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:43677)
0808 02:20:22.101042 (+ 3496us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.101120 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.102830 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.103397 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.103401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.103741 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:22.104030 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.104038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.105926 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.105928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.107724 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.107729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.108761 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.108763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.108935 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.109638 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:22.109650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:22.111953 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:22.114771 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:22.114782 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:22.114785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:22.115205 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:22.115524 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:22.115525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:22.115526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:22.115705 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:22.116263 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:22.116273 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:22.116476 (+ 203us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.116641 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.116938 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3445,"thread_start_us":129,"threads_started":1}
I20260808 02:20:22.118213 17442 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37135
I20260808 02:20:22.118346 17442 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:22.118541 17442 heartbeater.cc:511] Master 127.28.177.190:37135 requested a full tablet report, sending...
I20260808 02:20:22.119400 17178 ts_manager.cc:194] Registered new tserver with Master: cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333)
I20260808 02:20:22.120672 17178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:43677
I20260808 02:20:22.121723 17442 heartbeater.cc:503] Master 127.28.177.190:37135 was elected leader, sending a full tablet report...
I20260808 02:20:22.129050 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.120137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40333 (local address 127.0.0.1:55714)
0808 02:20:22.121084 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.121127 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.121204 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.125197 (+ 3993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.125201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.125204 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:22.125279 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.125281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.126718 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.126719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.127620 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.127623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.128670 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.128671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.128752 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.128764 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.128902 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.128980 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":115,"threads_started":1}
I20260808 02:20:22.129057 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.120233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.28.177.129:40333)
0808 02:20:22.124975 (+ 4742us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.124982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.124993 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:20:22.125034 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.125036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.125074 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.125170 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:22.125372 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.126604 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.127715 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.128559 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.128648 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.128710 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.128926 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.128971 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4700,"thread_start_us":110,"threads_started":1}
I20260808 02:20:22.129549 17340 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bbcf780114dfa9be2335f3e44002f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:21 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:55714
I20260808 02:20:22.130360 17452 tablet_replica.cc:330] stopping tablet replica
I20260808 02:20:22.130513 17452 raft_consensus.cc:2229] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:20:22.130611 17452 raft_consensus.cc:2258] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:22.130976 17452 ts_tablet_manager.cc:1905] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:22.133334 17452 ts_tablet_manager.cc:1918] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:22.133442 17452 log.cc:1196] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/609bbcf780114dfa9be2335f3e44002f
I20260808 02:20:22.133769 17452 ts_tablet_manager.cc:1939] T 609bbcf780114dfa9be2335f3e44002f P cd1850d533634d40a08bf5c1c1c097c5: Deleting consensus metadata
I20260808 02:20:22.134397 17163 catalog_manager.cc:4841] TS cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333): tablet 609bbcf780114dfa9be2335f3e44002f (table test-table [id=ec56ddccb54d47c0932248ddd3b20495]) successfully deleted
W20260808 02:20:22.205696 17448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:22.205914 17448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:22.205937 17448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:22.207245 17448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:22.207288 17448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:22.207351 17448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:20:22.209062 17448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:37933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=39629
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:22.209338 17448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:22.209589 17448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:22.215497 17465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.215570 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.215577 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.215718 17448 server_base.cc:1034] running on GCE node
I20260808 02:20:22.216015 17448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:22.216301 17448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:22.217526 17448 hybrid_clock.cc:648] HybridClock initialized: now 1786155622217502 us; error 41 us; skew 500 ppm
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:22.220218 17448 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:20:22.220718 17448 webserver.cc:460] Webserver started at http://127.28.177.130:39629/ using document root <none> and password file <none>
I20260808 02:20:22.221084 17448 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:22.221190 17448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:22.222697 17448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:20:22.223452 17473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.223655 17448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.223776 17448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "c64eb94be4ce4a1fa7df039741c78316"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "acf0233c3ad2d8f3a65a1cb1ffb8f48e"
server_key_iv: "c47e2748a1bea9a8ba3d2b37540e8da2"
server_key_version: "encryptionkey@0"
I20260808 02:20:22.224200 17448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:22.249782 17448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:22.252032 17448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:22.252234 17448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:22.252506 17448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:22.252959 17448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:22.253018 17448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.253060 17448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:22.253084 17448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.259392 17448 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:37933
I20260808 02:20:22.259447 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:37933 every 8 connection(s)
I20260808 02:20:22.259852 17448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:20:22.263756 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 17448
I20260808 02:20:22.264062 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:40079
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:37135
--webserver_port=40807
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:22.271601 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.260693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:53435 (local address 127.28.177.190:37135)
0808 02:20:22.260809 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.260812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.261050 (+ 238us) server_negotiation.cc:407] Connection header received
0808 02:20:22.261646 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.261647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.261661 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.261719 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:22.262737 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.263255 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.264799 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.265132 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.265205 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.267684 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:22.267686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:22.267687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:22.267696 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:22.269764 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.270346 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.270348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.270349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.270424 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.270749 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.270751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.270752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.270949 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:22.271037 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.271346 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.271459 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:20:22.271626 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.260596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:53435)
0808 02:20:22.260861 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.260910 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.261530 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.261793 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.261797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.262411 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:22.262624 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.262630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.263365 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.263367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.264687 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.264692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.265230 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.265231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.265407 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.266059 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:22.266072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:22.267557 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:22.269854 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:22.269862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:22.269864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:22.270207 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:22.270525 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:22.270526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:22.270526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:22.270663 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:22.271032 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:22.271038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:22.271190 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.271287 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.271422 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260808 02:20:22.272368 17633 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37135
I20260808 02:20:22.272466 17633 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:22.272681 17633 heartbeater.cc:511] Master 127.28.177.190:37135 requested a full tablet report, sending...
I20260808 02:20:22.273207 17178 ts_manager.cc:194] Registered new tserver with Master: c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933)
I20260808 02:20:22.273792 17178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:53435
W20260808 02:20:22.352885 17639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:22.353192 17639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:22.353224 17639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:22.354596 17639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:22.354643 17639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:22.354679 17639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:20:22.356212 17639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:40079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=40807
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:22.356441 17639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:22.356640 17639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:22.362224 17649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.362231 17647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.362412 17639 server_base.cc:1034] running on GCE node
W20260808 02:20:22.362291 17646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.362811 17639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:22.363180 17639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:22.364346 17639 hybrid_clock.cc:648] HybridClock initialized: now 1786155622364325 us; error 42 us; skew 500 ppm
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:22.367199 17639 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:20:22.367683 17639 webserver.cc:460] Webserver started at http://127.28.177.131:40807/ using document root <none> and password file <none>
I20260808 02:20:22.367945 17639 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:22.368033 17639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:22.369624 17639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:20:22.370294 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.370519 17639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.370635 17639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "d6eeae4d6d9c4387bff813269619cade"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "1fc0e50b291ff3f846f68912a9b07ca2"
server_key_iv: "014282258711cf27461ec85272ae1662"
server_key_version: "encryptionkey@0"
I20260808 02:20:22.370991 17639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:22.386857 17639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:22.389384 17639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:22.389564 17639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:22.389823 17639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:22.390177 17639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:22.390235 17639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.390276 17639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:22.390323 17639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.396529 17639 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:40079
I20260808 02:20:22.396672 17821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:40079 every 8 connection(s)
I20260808 02:20:22.397037 17639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:20:22.401192 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 17639
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:22.408677 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.397939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:48201 (local address 127.28.177.190:37135)
0808 02:20:22.398011 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.398013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.398275 (+ 262us) server_negotiation.cc:407] Connection header received
0808 02:20:22.398771 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.398772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.398785 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.398858 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:22.399829 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.400454 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.401995 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.402331 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.402390 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.405298 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:22.405300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:22.405301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:22.405308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:22.407142 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.407582 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.407584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.407584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.407643 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.407925 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.407926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.407926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.408069 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:22.408170 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.408486 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.408573 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:20:22.408939 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.397929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:48201)
0808 02:20:22.398119 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.398171 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.398676 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.398906 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.398910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.399514 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:22.399721 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.399725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.400574 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.400575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.401911 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.401915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.402523 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.402524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.402693 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.403431 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:22.403440 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:22.405201 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:22.407231 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:22.407240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:22.407243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:22.407488 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:22.407723 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:22.407724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:22.407724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:22.407847 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:22.408165 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:22.408174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:22.408332 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.408454 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.408636 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260808 02:20:22.409529 17822 heartbeater.cc:347] Connected to a master server at 127.28.177.190:37135
I20260808 02:20:22.409606 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.401669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.28.177.190:37135)
0808 02:20:22.401835 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.401837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.401895 (+ 58us) server_negotiation.cc:407] Connection header received
0808 02:20:22.401994 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.401995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.402008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.402195 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:22.403131 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.403776 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.404699 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.405012 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.405093 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.406478 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:22.406480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:22.406480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:22.406488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:22.408243 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.408591 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.408592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.408593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.408680 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:22.409053 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:22.409054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:22.409054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:22.409176 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:22.409374 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.409412 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.409500 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260808 02:20:22.409788 17822 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:22.410056 17822 heartbeater.cc:511] Master 127.28.177.190:37135 requested a full tablet report, sending...
I20260808 02:20:22.410473 17178 ts_manager.cc:194] Registered new tserver with Master: d6eeae4d6d9c4387bff813269619cade (127.28.177.131:40079)
I20260808 02:20:22.410805 17178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:48201
I20260808 02:20:22.421386 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:20:22.424816 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.421807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.28.177.190:37135)
0808 02:20:22.421908 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.421910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.422096 (+ 186us) server_negotiation.cc:407] Connection header received
0808 02:20:22.422172 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.422173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.422248 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.422341 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:20:22.422550 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.423162 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.423945 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.424211 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.424273 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.424472 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:20:22.424576 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:20:22.424641 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.424678 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.424701 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:20:22.425174 17178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:20:22.426739 17178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:20:22.429275 17340 tablet_service.cc:1467] Processing CreateTablet for tablet 1b00a817013341eabd747d5d179f2513 (DEFAULT_TABLE table=test-table [id=0b80ecf5751b4345851a15daeb30bc7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:22.429517 17340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b00a817013341eabd747d5d179f2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:22.430945 17289 tablet_bootstrap.cc:492] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Bootstrap starting.
I20260808 02:20:22.431296 17289 tablet_bootstrap.cc:654] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:22.431977 17289 tablet_bootstrap.cc:492] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: No bootstrap required, opened a new log
I20260808 02:20:22.432030 17289 ts_tablet_manager.cc:1397] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.432130 17289 raft_consensus.cc:348] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.432197 17289 raft_consensus.cc:374] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:22.432246 17289 raft_consensus.cc:729] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1850d533634d40a08bf5c1c1c097c5, State: Initialized, Role: FOLLOWER
I20260808 02:20:22.432286 17289 consensus_queue.cc:260] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.432358 17289 ts_tablet_manager.cc:1428] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.433578 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.428809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37933 (local address 127.0.0.1:57700)
0808 02:20:22.429050 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.429061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.429105 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.429758 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.429758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.429761 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:22.429799 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.429800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.431200 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.431201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.432362 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.432365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.433316 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.433318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.433379 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.433383 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.433446 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.433493 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":75,"thread_start_us":75,"threads_started":1}
I20260808 02:20:22.433593 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.429148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.28.177.130:37933)
0808 02:20:22.429364 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.429387 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.429399 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:20:22.429442 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.429444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.429460 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.429544 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:22.429857 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.431028 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.432459 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.433228 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.433290 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.433367 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.433465 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.433506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20260808 02:20:22.433609 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.428957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.28.177.131:40079)
0808 02:20:22.429380 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.429385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.429397 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:20:22.429439 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.429441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.429458 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.429544 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:22.429880 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.431074 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.432514 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.433237 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.433339 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.433395 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.433491 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.433532 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":106,"threads_started":1}
I20260808 02:20:22.433593 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.428808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:40079 (local address 127.0.0.1:49370)
0808 02:20:22.428897 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.428912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.428958 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.429558 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.429560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.429561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:22.429605 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.429606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.431166 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.431167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.432396 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.432400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.433347 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.433348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.433414 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.433418 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.433475 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.433520 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":99}
I20260808 02:20:22.434376 17542 tablet_service.cc:1467] Processing CreateTablet for tablet 1b00a817013341eabd747d5d179f2513 (DEFAULT_TABLE table=test-table [id=0b80ecf5751b4345851a15daeb30bc7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:22.434659 17733 tablet_service.cc:1467] Processing CreateTablet for tablet 1b00a817013341eabd747d5d179f2513 (DEFAULT_TABLE table=test-table [id=0b80ecf5751b4345851a15daeb30bc7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:22.434800 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b00a817013341eabd747d5d179f2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:22.434995 17733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b00a817013341eabd747d5d179f2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:22.438135 17836 tablet_bootstrap.cc:492] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Bootstrap starting.
I20260808 02:20:22.438263 17834 tablet_bootstrap.cc:492] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Bootstrap starting.
I20260808 02:20:22.438773 17836 tablet_bootstrap.cc:654] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:22.438840 17834 tablet_bootstrap.cc:654] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:22.439294 17836 log.cc:824] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:22.439337 17834 log.cc:824] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:22.440171 17834 tablet_bootstrap.cc:492] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: No bootstrap required, opened a new log
I20260808 02:20:22.440210 17836 tablet_bootstrap.cc:492] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: No bootstrap required, opened a new log
I20260808 02:20:22.440253 17834 ts_tablet_manager.cc:1397] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:20:22.440281 17836 ts_tablet_manager.cc:1397] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:20:22.441723 17834 raft_consensus.cc:348] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.441712 17836 raft_consensus.cc:348] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.441869 17836 raft_consensus.cc:374] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:22.441871 17834 raft_consensus.cc:374] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:22.441891 17836 raft_consensus.cc:729] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eeae4d6d9c4387bff813269619cade, State: Initialized, Role: FOLLOWER
I20260808 02:20:22.441922 17834 raft_consensus.cc:729] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64eb94be4ce4a1fa7df039741c78316, State: Initialized, Role: FOLLOWER
I20260808 02:20:22.442046 17836 consensus_queue.cc:260] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.442046 17834 consensus_queue.cc:260] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.442371 17633 heartbeater.cc:503] Master 127.28.177.190:37135 was elected leader, sending a full tablet report...
I20260808 02:20:22.442363 17836 ts_tablet_manager.cc:1428] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:20:22.442373 17822 heartbeater.cc:503] Master 127.28.177.190:37135 was elected leader, sending a full tablet report...
I20260808 02:20:22.442656 17834 ts_tablet_manager.cc:1428] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260808 02:20:22.510560 17634 tablet.cc:2369] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:20:22.593531 17449 raft_consensus.cc:482] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:20:22.593704 17449 raft_consensus.cc:504] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.594038 17449 leader_election.cc:288] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933), d6eeae4d6d9c4387bff813269619cade (127.28.177.131:40079)
I20260808 02:20:22.598532 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.594451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:42591 (local address 127.28.177.130:37933)
0808 02:20:22.594513 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.594515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.594782 (+ 267us) server_negotiation.cc:407] Connection header received
0808 02:20:22.594827 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.594828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.594837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.594880 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:22.595054 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.595857 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.597503 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.598096 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.598184 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.598195 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.598399 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.598436 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:20:22.598567 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.594310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:40079 (local address 127.28.177.129:38731)
0808 02:20:22.594433 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.594448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.594517 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.594706 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.594707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.594709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:22.595127 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.595163 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.596022 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.596022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.597399 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.597403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.598319 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.598320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.598411 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.598414 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.598466 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.598501 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":12}
I20260808 02:20:22.598598 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.594419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:38731 (local address 127.28.177.131:40079)
0808 02:20:22.594489 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.594491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.594509 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:20:22.594589 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.594591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.594604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.594687 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:22.595263 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.595944 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.597503 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.598218 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.598282 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.598289 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.598491 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.598534 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:20:22.598574 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.594312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37933 (local address 127.28.177.129:42591)
0808 02:20:22.594695 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.594704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.594767 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:22.594884 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:22.594885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:22.594886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:22.594957 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.594959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.595947 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.595948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.597399 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:22.597403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.598207 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.598208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:22.598307 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.598313 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:20:22.598385 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:20:22.598436 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":302,"thread_start_us":110,"threads_started":1}
I20260808 02:20:22.599287 17759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b00a817013341eabd747d5d179f2513" candidate_uuid: "cd1850d533634d40a08bf5c1c1c097c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6eeae4d6d9c4387bff813269619cade" is_pre_election: true
I20260808 02:20:22.599350 17571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b00a817013341eabd747d5d179f2513" candidate_uuid: "cd1850d533634d40a08bf5c1c1c097c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64eb94be4ce4a1fa7df039741c78316" is_pre_election: true
I20260808 02:20:22.599475 17759 raft_consensus.cc:2454] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1850d533634d40a08bf5c1c1c097c5 in term 0.
I20260808 02:20:22.599560 17571 raft_consensus.cc:2454] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1850d533634d40a08bf5c1c1c097c5 in term 0.
I20260808 02:20:22.599794 17286 leader_election.cc:302] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64eb94be4ce4a1fa7df039741c78316, cd1850d533634d40a08bf5c1c1c097c5; no voters:
I20260808 02:20:22.599990 17449 raft_consensus.cc:2781] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:20:22.600059 17449 raft_consensus.cc:482] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:20:22.600077 17449 raft_consensus.cc:3037] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:22.600945 17449 raft_consensus.cc:504] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
I20260808 02:20:22.601131 17449 leader_election.cc:288] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [CANDIDATE]: Term 1 election: Requested vote from peers c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933), d6eeae4d6d9c4387bff813269619cade (127.28.177.131:40079)
I20260808 02:20:22.601310 17571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b00a817013341eabd747d5d179f2513" candidate_uuid: "cd1850d533634d40a08bf5c1c1c097c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64eb94be4ce4a1fa7df039741c78316"
I20260808 02:20:22.601401 17571 raft_consensus.cc:3037] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:22.601434 17759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b00a817013341eabd747d5d179f2513" candidate_uuid: "cd1850d533634d40a08bf5c1c1c097c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6eeae4d6d9c4387bff813269619cade"
I20260808 02:20:22.601567 17759 raft_consensus.cc:3037] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:22.602233 17571 raft_consensus.cc:2454] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1850d533634d40a08bf5c1c1c097c5 in term 1.
I20260808 02:20:22.602458 17286 leader_election.cc:302] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64eb94be4ce4a1fa7df039741c78316, cd1850d533634d40a08bf5c1c1c097c5; no voters:
I20260808 02:20:22.602453 17759 raft_consensus.cc:2454] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1850d533634d40a08bf5c1c1c097c5 in term 1.
I20260808 02:20:22.602564 17449 raft_consensus.cc:2781] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:22.602733 17449 raft_consensus.cc:686] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 1 LEADER]: Becoming Leader. State: Replica: cd1850d533634d40a08bf5c1c1c097c5, State: Running, Role: LEADER
I20260808 02:20:22.602836 17449 consensus_queue.cc:237] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } }
W20260808 02:20:22.602983 17444 tablet.cc:2369] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:20:22.603866 17178 catalog_manager.cc:5495] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "cd1850d533634d40a08bf5c1c1c097c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd1850d533634d40a08bf5c1c1c097c5" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40333 } health_report { overall_health: HEALTHY } } }
I20260808 02:20:22.619962 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.617505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.28.177.129:40333)
0808 02:20:22.617598 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:22.617600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:22.617611 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:20:22.617655 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:22.617656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:22.617681 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:22.617762 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:20:22.617889 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.618414 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.619184 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:22.619385 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:22.619468 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:22.619598 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:20:22.619709 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:20:22.619772 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:22.619810 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:20:22.619834 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:20:22.621877 17571 raft_consensus.cc:1261] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Refusing update from remote peer cd1850d533634d40a08bf5c1c1c097c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:22.621877 17759 raft_consensus.cc:1261] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Refusing update from remote peer cd1850d533634d40a08bf5c1c1c097c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:22.622278 17449 consensus_queue.cc:1035] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6eeae4d6d9c4387bff813269619cade" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:22.622427 17845 consensus_queue.cc:1035] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64eb94be4ce4a1fa7df039741c78316" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:22.625167 17850 mvcc.cc:205] Tried to move back new op lower bound from 7316093430256435200 to 7316093430182449152. Current Snapshot: MvccSnapshot[applied={T|T < 7316093430256435200}]
I20260808 02:20:22.626138 17851 mvcc.cc:205] Tried to move back new op lower bound from 7316093430256435200 to 7316093430182449152. Current Snapshot: MvccSnapshot[applied={T|T < 7316093430256435200}]
I20260808 02:20:22.627692 17176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:22.627806 17176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:22.628648 17176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Sending DeleteTablet for 3 replicas of tablet 1b00a817013341eabd747d5d179f2513
I20260808 02:20:22.628996 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b00a817013341eabd747d5d179f2513 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:22 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:57700
I20260808 02:20:22.629052 17340 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b00a817013341eabd747d5d179f2513 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:22 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:55714
I20260808 02:20:22.629112 17733 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b00a817013341eabd747d5d179f2513 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:22 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:49370
I20260808 02:20:22.629148 17452 tablet_replica.cc:330] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: stopping tablet replica
I20260808 02:20:22.629231 17452 raft_consensus.cc:2229] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:20:22.629267 17862 tablet_replica.cc:330] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: stopping tablet replica
I20260808 02:20:22.629312 17452 raft_consensus.cc:2258] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:22.629325 17864 tablet_replica.cc:330] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: stopping tablet replica
I20260808 02:20:22.629364 17862 raft_consensus.cc:2229] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:20:22.629415 17864 raft_consensus.cc:2229] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:20:22.629465 17862 raft_consensus.cc:2258] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:22.629525 17864 raft_consensus.cc:2258] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:22.629914 17862 ts_tablet_manager.cc:1905] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:22.629925 17452 ts_tablet_manager.cc:1905] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:22.629976 17864 ts_tablet_manager.cc:1905] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:22.631589 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 17235
I20260808 02:20:22.637372 17864 ts_tablet_manager.cc:1918] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:22.637490 17864 log.cc:1196] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/1b00a817013341eabd747d5d179f2513
I20260808 02:20:22.637784 17864 ts_tablet_manager.cc:1939] T 1b00a817013341eabd747d5d179f2513 P d6eeae4d6d9c4387bff813269619cade: Deleting consensus metadata
I20260808 02:20:22.638239 17862 ts_tablet_manager.cc:1918] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:22.638327 17862 log.cc:1196] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/1b00a817013341eabd747d5d179f2513
I20260808 02:20:22.638320 17161 catalog_manager.cc:4841] TS d6eeae4d6d9c4387bff813269619cade (127.28.177.131:40079): tablet 1b00a817013341eabd747d5d179f2513 (table test-table [id=0b80ecf5751b4345851a15daeb30bc7f]) successfully deleted
I20260808 02:20:22.638626 17862 ts_tablet_manager.cc:1939] T 1b00a817013341eabd747d5d179f2513 P c64eb94be4ce4a1fa7df039741c78316: Deleting consensus metadata
I20260808 02:20:22.639075 17163 catalog_manager.cc:4841] TS c64eb94be4ce4a1fa7df039741c78316 (127.28.177.130:37933): tablet 1b00a817013341eabd747d5d179f2513 (table test-table [id=0b80ecf5751b4345851a15daeb30bc7f]) successfully deleted
W20260808 02:20:22.644752 17163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:40333 (error 108)
I20260808 02:20:22.645439 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:22.645268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40333 (local address 127.0.0.1:55732)
0808 02:20:22.645331 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.645376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:20:22.645501 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 17448
W20260808 02:20:22.645784 17163 catalog_manager.cc:4584] TS cd1850d533634d40a08bf5c1c1c097c5 (127.28.177.129:40333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b00a817013341eabd747d5d179f2513: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40333: connect: Connection refused (error 111)
I20260808 02:20:22.658087 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 17639
I20260808 02:20:22.672201 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 17136
I20260808 02:20:22.681792 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37135
--webserver_interface=127.28.177.190
--webserver_port=33027
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:37135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:22.771174 17872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:22.771399 17872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:22.771422 17872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:22.772614 17872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:22.772663 17872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:22.772677 17872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:22.772691 17872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:22.772702 17872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:22.774392 17872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:37135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:37135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:22.774677 17872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:22.774925 17872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:22.780709 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.780727 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.780725 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.781035 17872 server_base.cc:1034] running on GCE node
I20260808 02:20:22.781368 17872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:22.781677 17872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:22.782848 17872 hybrid_clock.cc:648] HybridClock initialized: now 1786155622782826 us; error 37 us; skew 500 ppm
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:22.785539 17872 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:22.786016 17872 webserver.cc:460] Webserver started at http://127.28.177.190:33027/ using document root <none> and password file <none>
I20260808 02:20:22.786223 17872 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:22.786273 17872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:22.787649 17872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.788372 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.788625 17872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.788745 17872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "f90d689397ef4fafbe822da41ae70cdd"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "ad6cfcea4d432b1e9a5d60030f0da12f"
server_key_iv: "66142a7e1f2420fd63e7c6cb21dcefaa"
server_key_version: "encryptionkey@0"
I20260808 02:20:22.789222 17872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:22.801991 17872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:22.804776 17872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:22.805069 17872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:22.810423 17872 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:37135
I20260808 02:20:22.810465 17961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:37135 every 8 connection(s)
I20260808 02:20:22.810911 17872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:22.812824 17962 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:20:22.813612 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap starting.
I20260808 02:20:22.818101 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 17872
I20260808 02:20:22.818452 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:40333
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:37135
--webserver_port=38745
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:22.824047 17962 log.cc:824] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:22.826773 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:20:22.827154 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd: Bootstrap complete.
I20260808 02:20:22.829883 17962 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:22.830181 17962 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90d689397ef4fafbe822da41ae70cdd, State: Initialized, Role: FOLLOWER
I20260808 02:20:22.830394 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:22.830483 17962 raft_consensus.cc:388] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:22.830549 17962 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:22.830617 17962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:20:22.831781 17962 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:22.831924 17962 leader_election.cc:302] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90d689397ef4fafbe822da41ae70cdd; no voters:
I20260808 02:20:22.832166 17962 leader_election.cc:288] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:20:22.832209 17967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:20:22.832333 17967 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [term 3 LEADER]: Becoming Leader. State: Replica: f90d689397ef4fafbe822da41ae70cdd, State: Running, Role: LEADER
I20260808 02:20:22.832444 17967 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } }
I20260808 02:20:22.832485 17962 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:22.832748 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90d689397ef4fafbe822da41ae70cdd. Latest consensus state: current_term: 3 leader_uuid: "f90d689397ef4fafbe822da41ae70cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } } }
I20260808 02:20:22.832820 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:22.832996 17968 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f90d689397ef4fafbe822da41ae70cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d689397ef4fafbe822da41ae70cdd" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 37135 } } }
I20260808 02:20:22.833105 17968 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d689397ef4fafbe822da41ae70cdd [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:22.833437 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:22.834483 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:22.834610 17977 catalog_manager.cc:1259] Loaded cluster ID: 1e3448a43bb044ab8ff14eb57adaf52b
I20260808 02:20:22.834653 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 02:20:22.834955 17977 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0fe17f89ad google::LogMessage::Fail() at ??:0
@ 0x7f0fe17fb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0fe17f84de google::LogMessage::Flush() at ??:0
@ 0x7f0fe17f9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0fe971d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0fe97268b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0fe36c60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0fe36bb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0fe451f6db start_thread at ??:0
@ 0x7f0fe02f661f clone at ??:0
W20260808 02:20:22.919368 17965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:22.919593 17965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:22.919620 17965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:22.920922 17965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:22.920967 17965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:22.920996 17965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:20:22.922602 17965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:40333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=38745
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:22.922889 17965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:22.923151 17965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:22.929117 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.929169 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:22.929121 18027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:22.929955 17965 server_base.cc:1034] running on GCE node
I20260808 02:20:22.930274 17965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:22.930540 17965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:22.931721 17965 hybrid_clock.cc:648] HybridClock initialized: now 1786155622931702 us; error 33 us; skew 500 ppm
Aug 08 02:20:22 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:22.934624 17965 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:20:22.935127 17965 webserver.cc:460] Webserver started at http://127.28.177.129:38745/ using document root <none> and password file <none>
I20260808 02:20:22.935398 17965 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:22.935470 17965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:22.936983 17965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.937881 18036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.938130 17965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:22.938251 17965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "cd1850d533634d40a08bf5c1c1c097c5"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "5e5d5655a4e48d24108c2466d682565f"
server_key_iv: "23f5a652d02d3d33b8ddf7036b9c0840"
server_key_version: "encryptionkey@0"
I20260808 02:20:22.938627 17965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:22.955600 17965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:22.958227 17965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:22.958417 17965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:22.958694 17965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:22.959221 18047 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260808 02:20:22.961529 18047 ts_tablet_manager.cc:1852] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 02:20:22.961967 18047 ts_tablet_manager.cc:1905] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:22.964977 18047 ts_tablet_manager.cc:1918] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:22.965111 18047 log.cc:1196] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/1b00a817013341eabd747d5d179f2513
I20260808 02:20:22.965448 18047 ts_tablet_manager.cc:1939] T 1b00a817013341eabd747d5d179f2513 P cd1850d533634d40a08bf5c1c1c097c5: Deleting consensus metadata
I20260808 02:20:22.965732 17965 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 02:20:22.965803 17965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260808 02:20:22.965854 17965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:22.965883 17965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:22.973935 17965 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:40333
I20260808 02:20:22.974031 18194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:40333 every 8 connection(s)
I20260808 02:20:22.974361 17965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:20:22.975178 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 17965
I20260808 02:20:22.975466 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:37933
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:37135
--webserver_port=39629
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:23.082629 18198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:23.082901 18198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:23.082947 18198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:23.084177 18198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:23.084246 18198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:23.084293 18198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:20:23.085911 18198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:37933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=39629
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:23.086241 18198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:23.086488 18198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:23.093099 18205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:23.093164 18212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:23.093111 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:23.093573 18198 server_base.cc:1034] running on GCE node
I20260808 02:20:23.093808 18198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:23.094138 18198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:23.095323 18198 hybrid_clock.cc:648] HybridClock initialized: now 1786155623095298 us; error 45 us; skew 500 ppm
Aug 08 02:20:23 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155623, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:23.098448 18198 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:20:23.099037 18198 webserver.cc:460] Webserver started at http://127.28.177.130:39629/ using document root <none> and password file <none>
I20260808 02:20:23.099371 18198 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:23.099450 18198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:23.101114 18198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:20:23.101895 18224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:23.102162 18198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:20:23.102281 18198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "c64eb94be4ce4a1fa7df039741c78316"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "acf0233c3ad2d8f3a65a1cb1ffb8f48e"
server_key_iv: "c47e2748a1bea9a8ba3d2b37540e8da2"
server_key_version: "encryptionkey@0"
I20260808 02:20:23.102790 18198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:23.121526 18198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:23.124554 18198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:23.124730 18198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:23.125056 18198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:23.125468 18198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:23.125519 18198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:23.125556 18198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:23.125588 18198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260808 02:20:23.127621 18197 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:20:22.975219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:39283)
0808 02:20:22.975455 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:22.975513 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:22.976271 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:23.127337 (+151066us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
W20260808 02:20:23.127856 18195 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:37135 (0 consecutive failures): Network error: Failed to ping master at 127.28.177.190:37135: Client connection negotiation failed: client connection to 127.28.177.190:37135: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:20:23.128273 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.128090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:57405)
0808 02:20:23.128168 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.128232 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 02:20:23.128683 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.128536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:54301)
0808 02:20:23.128613 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.128653 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:20:23.128816 18195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:20:23.132671 18198 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:37933
I20260808 02:20:23.132758 18385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:37933 every 8 connection(s)
I20260808 02:20:23.133157 18198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:20:23.133960 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 18198
I20260808 02:20:23.134289 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:40079
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:37135
--webserver_port=40807
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:20:23.134451 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.133974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:47351)
0808 02:20:23.134165 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.134252 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":119,"threads_started":1}
W20260808 02:20:23.134945 18386 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:37135 (0 consecutive failures): Network error: Failed to ping master at 127.28.177.190:37135: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
I20260808 02:20:23.135416 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.135156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:36085)
0808 02:20:23.135328 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.135370 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260808 02:20:23.135744 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.135658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:43387)
0808 02:20:23.135704 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.135728 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 02:20:23.135875 18386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:20:23.222872 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:23.223166 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:23.223212 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:23.224460 18389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:23.224536 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:23.224584 18389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:20:23.226166 18389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:40079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=40807
--tserver_master_addrs=127.28.177.190:37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:23.226469 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:23.226730 18389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:23.232465 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:23.232458 18400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:23.232460 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:23.232566 18389 server_base.cc:1034] running on GCE node
I20260808 02:20:23.232950 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:23.233173 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:23.234345 18389 hybrid_clock.cc:648] HybridClock initialized: now 1786155623234334 us; error 40 us; skew 500 ppm
Aug 08 02:20:23 dist-test-slave-3h5h krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155623, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:23.237249 18389 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:20:23.237725 18389 webserver.cc:460] Webserver started at http://127.28.177.131:40807/ using document root <none> and password file <none>
I20260808 02:20:23.237968 18389 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:23.238039 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:23.239482 18389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:23.240157 18407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:23.240417 18389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:23.240545 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "d6eeae4d6d9c4387bff813269619cade"
format_stamp: "Formatted at 2026-08-08 02:20:21 on dist-test-slave-3h5h"
server_key: "1fc0e50b291ff3f846f68912a9b07ca2"
server_key_iv: "014282258711cf27461ec85272ae1662"
server_key_version: "encryptionkey@0"
I20260808 02:20:23.241142 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:23.271349 18389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:23.274297 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:23.274500 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:23.274814 18389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:23.275205 18389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:23.275266 18389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:23.275313 18389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:23.275364 18389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:23.282456 18389 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:40079
I20260808 02:20:23.282508 18565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:40079 every 8 connection(s)
I20260808 02:20:23.282915 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:20:23.284200 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.283633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:41095)
0808 02:20:23.283833 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.283924 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
W20260808 02:20:23.284729 18566 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:37135 (0 consecutive failures): Network error: Failed to ping master at 127.28.177.190:37135: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
I20260808 02:20:23.285167 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.284970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:59329)
0808 02:20:23.285112 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.285145 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:20:23.285669 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:23.285528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:36949)
0808 02:20:23.285608 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:23.285642 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 02:20:23.285941 18566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:20:23.291371 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 18389
I20260808 02:20:24.129546 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:24.129241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:42589)
0808 02:20:24.129419 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:24.129478 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260808 02:20:24.136457 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:24.136216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:59619)
0808 02:20:24.136366 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:24.136403 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260808 02:20:24.286618 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:24.286353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:43873)
0808 02:20:24.286499 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:24.286548 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260808 02:20:25.130520 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:25.130274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:45753)
0808 02:20:25.130427 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:25.130473 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260808 02:20:25.137238 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:25.137000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:58197)
0808 02:20:25.137135 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:25.137193 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260808 02:20:25.287446 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:25.287215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:56929)
0808 02:20:25.287353 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:25.287395 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260808 02:20:26.131392 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:26.131135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:55707)
0808 02:20:26.131299 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:26.131344 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260808 02:20:26.138041 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:26.137779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:60081)
0808 02:20:26.137933 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:26.137985 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260808 02:20:26.288220 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:26.287935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:44843)
0808 02:20:26.288124 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:26.288171 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260808 02:20:27.132210 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:27.131919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:46151)
0808 02:20:27.132112 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:27.132159 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260808 02:20:27.138947 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:27.138685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:41205)
0808 02:20:27.138854 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:27.138899 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260808 02:20:27.289141 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:27.288818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:35089)
0808 02:20:27.289024 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:27.289084 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":115,"threads_started":1}
I20260808 02:20:28.133219 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:28.132909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:50123)
0808 02:20:28.133096 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:28.133168 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260808 02:20:28.139808 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:28.139561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:47911)
0808 02:20:28.139716 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:28.139759 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260808 02:20:28.289942 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:28.289670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:55233)
0808 02:20:28.289847 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:28.289889 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260808 02:20:29.134164 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:29.133842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:51343)
0808 02:20:29.134011 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:29.134057 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260808 02:20:29.140751 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:29.140402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:45893)
0808 02:20:29.140626 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:29.140685 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260808 02:20:29.290817 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:29.290580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:43803)
0808 02:20:29.290728 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:29.290767 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260808 02:20:30.134887 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:30.134667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:52697)
0808 02:20:30.134806 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:30.134844 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260808 02:20:30.141685 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:30.141366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:44169)
0808 02:20:30.141592 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:30.141640 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":64,"threads_started":1}
I20260808 02:20:30.291548 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:30.291278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:36895)
0808 02:20:30.291454 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:30.291493 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260808 02:20:31.135677 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:31.135454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:41819)
0808 02:20:31.135594 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:31.135636 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260808 02:20:31.142524 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:31.142255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:36571)
0808 02:20:31.142413 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:31.142460 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260808 02:20:31.292290 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:31.292080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:43545)
0808 02:20:31.292211 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:31.292242 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260808 02:20:32.136530 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:32.136216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:35359)
0808 02:20:32.136412 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:32.136485 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":145,"threads_started":1}
I20260808 02:20:32.143365 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:32.143100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:43691)
0808 02:20:32.143272 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:32.143306 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260808 02:20:32.293020 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:32.292762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:42737)
0808 02:20:32.292929 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:32.292966 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":117,"threads_started":1}
I20260808 02:20:33.137393 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:33.137114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:53707)
0808 02:20:33.137302 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:33.137346 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260808 02:20:33.144105 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:33.143873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:50971)
0808 02:20:33.144025 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:33.144060 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260808 02:20:33.293823 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:33.293533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:40381)
0808 02:20:33.293716 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:33.293771 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260808 02:20:34.138266 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:34.137971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:41515)
0808 02:20:34.138168 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:34.138220 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260808 02:20:34.144948 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:34.144642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:47145)
0808 02:20:34.144813 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:34.144901 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260808 02:20:34.294711 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:34.294428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:56469)
0808 02:20:34.294607 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:34.294657 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260808 02:20:35.139041 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:35.138811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:46847)
0808 02:20:35.138953 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:35.138992 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260808 02:20:35.145779 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:35.145510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:54999)
0808 02:20:35.145686 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:35.145731 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260808 02:20:35.295640 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:35.295329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:47903)
0808 02:20:35.295530 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:35.295588 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260808 02:20:36.139886 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:36.139652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:46845)
0808 02:20:36.139792 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:36.139835 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260808 02:20:36.146654 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:36.146364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:51069)
0808 02:20:36.146519 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:36.146576 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260808 02:20:36.296607 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:36.296356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:51859)
0808 02:20:36.296510 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:36.296554 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260808 02:20:37.140703 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:37.140423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:56757)
0808 02:20:37.140605 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:37.140649 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260808 02:20:37.147424 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:37.147204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:42811)
0808 02:20:37.147335 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:37.147374 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260808 02:20:37.301316 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:37.301035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:54309)
0808 02:20:37.301195 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:37.301250 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260808 02:20:38.141817 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:38.141487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.129:43579)
0808 02:20:38.141699 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:38.141766 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260808 02:20:38.148389 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:38.148135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.130:39775)
0808 02:20:38.148303 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:38.148343 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260808 02:20:38.296172 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 17965
I20260808 02:20:38.303382 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:38.303142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:37135 (local address 127.28.177.131:57031)
0808 02:20:38.303285 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:38.303332 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:37135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260808 02:20:38.308933 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 18198
I20260808 02:20:38.319147 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 18389
2026-08-08T02:20:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18794 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:20:38 dist-test-slave-3h5h krb5kdc[18637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:20:38 dist-test-slave-3h5h krb5kdc[18637](info): set up 2 sockets
Aug 08 02:20:38 dist-test-slave-3h5h krb5kdc[18637](info): commencing operation
krb5kdc: starting...
W20260808 02:20:39.581709 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:20:39 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:20:39.720999 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41243
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:39.808020 18653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:39.808264 18653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:39.808303 18653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:39.809544 18653 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:20:39.809619 18653 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:20:39.809646 18653 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:20:39.809676 18653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:20:39.809702 18653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:39.809734 18653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:39.809762 18653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:20:39.809791 18653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:20:39.811314 18653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:39.811612 18653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:39.811867 18653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:39.817682 18659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:39.817739 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:39.817740 18653 server_base.cc:1034] running on GCE node
W20260808 02:20:39.817706 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:39.818298 18653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:39.818608 18653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:39.819818 18653 hybrid_clock.cc:648] HybridClock initialized: now 1786155639819795 us; error 44 us; skew 500 ppm
Aug 08 02:20:39 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:39.822624 18653 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:20:39.823182 18653 webserver.cc:460] Webserver started at http://127.28.177.190:33287/ using document root <none> and password file <none>
I20260808 02:20:39.823459 18653 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:39.823534 18653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:39.823686 18653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:39.824766 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "d6c399d2fc51444bb3a30a5dddca2799"
format_stamp: "Formatted at 2026-08-08 02:20:39 on dist-test-slave-3h5h"
server_key: "065efddd121fb6443974527231466a4f"
server_key_iv: "cb19085afcf224426143b56ba39c684a"
server_key_version: "encryptionkey@0"
I20260808 02:20:39.825209 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "d6c399d2fc51444bb3a30a5dddca2799"
format_stamp: "Formatted at 2026-08-08 02:20:39 on dist-test-slave-3h5h"
server_key: "065efddd121fb6443974527231466a4f"
server_key_iv: "cb19085afcf224426143b56ba39c684a"
server_key_version: "encryptionkey@0"
I20260808 02:20:39.826606 18653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:39.827474 18677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:39.827689 18653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:39.827800 18653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "d6c399d2fc51444bb3a30a5dddca2799"
format_stamp: "Formatted at 2026-08-08 02:20:39 on dist-test-slave-3h5h"
server_key: "065efddd121fb6443974527231466a4f"
server_key_iv: "cb19085afcf224426143b56ba39c684a"
server_key_version: "encryptionkey@0"
I20260808 02:20:39.827876 18653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:39.848750 18653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:39.851212 18653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:39.851436 18653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:39.856812 18653 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:41243
I20260808 02:20:39.856889 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:41243 every 8 connection(s)
I20260808 02:20:39.857434 18653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:20:39.857710 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 18653
I20260808 02:20:39.857846 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:20:39.858095 29382 external_mini_cluster.cc:1428] Setting key 2c74d7f738359c6e135e78581b6c4065
I20260808 02:20:39.858698 18761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:39.862200 18761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap starting.
I20260808 02:20:39.862838 18761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:39.863193 18761 log.cc:824] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:39.864044 18761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: No bootstrap required, opened a new log
Aug 08 02:20:39 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:39.866671 18761 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:20:39.866842 18761 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:39.866890 18761 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c399d2fc51444bb3a30a5dddca2799, State: Initialized, Role: FOLLOWER
I20260808 02:20:39.867033 18761 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:20:39.867131 18761 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:20:39.867179 18761 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:20:39.867230 18761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:39.867991 18761 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:20:39.868126 18761 leader_election.cc:302] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c399d2fc51444bb3a30a5dddca2799; no voters:
I20260808 02:20:39.868364 18761 leader_election.cc:288] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:20:39.868398 18769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:39.868515 18769 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 LEADER]: Becoming Leader. State: Replica: d6c399d2fc51444bb3a30a5dddca2799, State: Running, Role: LEADER
I20260808 02:20:39.868604 18769 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:20:39.868698 18761 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:20:39.868975 18769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c399d2fc51444bb3a30a5dddca2799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } } }
I20260808 02:20:39.869124 18769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: This master's current role is: LEADER
I20260808 02:20:39.869136 18770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c399d2fc51444bb3a30a5dddca2799. Latest consensus state: current_term: 1 leader_uuid: "d6c399d2fc51444bb3a30a5dddca2799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } } }
I20260808 02:20:39.869375 18770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: This master's current role is: LEADER
W20260808 02:20:39.870795 18788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:20:39.870841 18788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:20:39.870903 18777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:20:39.871101 18777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:20:39.871748 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:39.858776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.28.177.190:41243)
0808 02:20:39.858984 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:39.858995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:39.859030 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:20:39.859637 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:39.859647 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:39.859725 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:39.859882 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:39.860708 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:39.862102 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:39.863011 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:39.863619 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:39.863735 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:39.867020 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:39.867026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:39.867026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:39.867051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:39.869590 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:39.870091 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:39.870095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:39.870095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:39.870226 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:39.870606 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:39.870607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:39.870607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:39.871010 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:39.871255 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:39.871333 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:20:39.871500 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260808 02:20:39.872987 18777 catalog_manager.cc:1347] Generated new cluster ID: 2731e109ee424054abd66c61fdff3725
I20260808 02:20:39.873027 18777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:20:39.888530 18777 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:20:39.889181 18777 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:20:39.892040 18777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Generated new TSK 0
I20260808 02:20:39.892230 18777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:20:39.929173 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:41243
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:40.025168 18795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:40.025413 18795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:40.025444 18795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:40.026777 18795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:40.026825 18795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:40.026854 18795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:20:40.028432 18795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:40.028693 18795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:40.029011 18795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:40.034679 18801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:40.034691 18805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:40.034691 18803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:40.034906 18795 server_base.cc:1034] running on GCE node
I20260808 02:20:40.035162 18795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:40.035444 18795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:40.036633 18795 hybrid_clock.cc:648] HybridClock initialized: now 1786155640036607 us; error 39 us; skew 500 ppm
Aug 08 02:20:40 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:40.039525 18795 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:20:40.040077 18795 webserver.cc:460] Webserver started at http://127.28.177.129:36679/ using document root <none> and password file <none>
I20260808 02:20:40.040342 18795 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:40.040413 18795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:40.040570 18795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:40.041756 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "28b749f373d04a7ca94fbe58035ed119"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "61f971c4647110d527e705f31f3f1c50"
server_key_iv: "752c4e05ba34d7fc5fb1833aee7455eb"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.042143 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "28b749f373d04a7ca94fbe58035ed119"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "61f971c4647110d527e705f31f3f1c50"
server_key_iv: "752c4e05ba34d7fc5fb1833aee7455eb"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.043643 18795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:20:40.044539 18814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.044739 18795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:20:40.044869 18795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "28b749f373d04a7ca94fbe58035ed119"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "61f971c4647110d527e705f31f3f1c50"
server_key_iv: "752c4e05ba34d7fc5fb1833aee7455eb"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.044966 18795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:40.065500 18795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:40.068024 18795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:40.068209 18795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:40.068487 18795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:40.068948 18795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:40.069005 18795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.069051 18795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:40.069079 18795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.075347 18795 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:33409
I20260808 02:20:40.075471 18979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:33409 every 8 connection(s)
I20260808 02:20:40.075785 18795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:20:40.076787 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 18795
I20260808 02:20:40.076932 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:20:40.077183 29382 external_mini_cluster.cc:1428] Setting key 4bd35bee4e5b3aff0dcd2fd93515367a
Aug 08 02:20:40 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:20:40.086886 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.076686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:36153 (local address 127.28.177.190:41243)
0808 02:20:40.076739 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.076741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.077023 (+ 282us) server_negotiation.cc:407] Connection header received
0808 02:20:40.077788 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.077790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.077805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.077871 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:40.079020 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.079598 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.080670 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.080994 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.081078 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.083703 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:40.083705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:40.083705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:40.083721 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:40.085343 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.085802 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.085803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.085804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.085866 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.086131 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.086132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.086133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.086277 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:40.086355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.086670 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.086785 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:20:40.087077 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.076635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:36153)
0808 02:20:40.076821 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.076917 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.077667 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.078073 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.078077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.078680 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:40.078886 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.078891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.079739 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.079741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.080555 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.080557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.081109 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.081110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.081239 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.081809 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:40.081819 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:40.083587 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:40.085454 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:40.085464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:40.085466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:40.085709 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:40.085942 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:40.085943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:40.085943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:40.086059 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:40.086357 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:40.086365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:40.086519 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.086632 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.086811 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260808 02:20:40.088091 18980 heartbeater.cc:347] Connected to a master server at 127.28.177.190:41243
I20260808 02:20:40.088299 18980 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:40.088557 18980 heartbeater.cc:511] Master 127.28.177.190:41243 requested a full tablet report, sending...
I20260808 02:20:40.089284 18704 ts_manager.cc:194] Registered new tserver with Master: 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409)
I20260808 02:20:40.089936 18704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:36153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:20:40.109200 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:41243
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:40.199925 18987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:40.200210 18987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:40.200253 18987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:40.201447 18987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:40.201529 18987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:40.201574 18987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:20:40.203104 18987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:40.203382 18987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:40.203630 18987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:40.209138 18993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:40.209198 18996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:40.209321 18987 server_base.cc:1034] running on GCE node
W20260808 02:20:40.209415 18994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:40.209679 18987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:40.209975 18987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:40.211185 18987 hybrid_clock.cc:648] HybridClock initialized: now 1786155640211157 us; error 31 us; skew 500 ppm
Aug 08 02:20:40 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:40.214177 18987 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:20:40.214742 18987 webserver.cc:460] Webserver started at http://127.28.177.130:33987/ using document root <none> and password file <none>
I20260808 02:20:40.215035 18987 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:40.215137 18987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:40.215330 18987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:40.216621 18987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "7004854aedc24e9e93065d0ee39fb6b3"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "ec715513df9c2f82869b44fce5fb8a7a"
server_key_iv: "74c97776436ec2bfef108b6ff17fce34"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.217121 18987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "7004854aedc24e9e93065d0ee39fb6b3"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "ec715513df9c2f82869b44fce5fb8a7a"
server_key_iv: "74c97776436ec2bfef108b6ff17fce34"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.218752 18987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:20:40.219653 19006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.219923 18987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:40.220033 18987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "7004854aedc24e9e93065d0ee39fb6b3"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "ec715513df9c2f82869b44fce5fb8a7a"
server_key_iv: "74c97776436ec2bfef108b6ff17fce34"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.220165 18987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:40.242270 18987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:40.244457 18987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:40.244639 18987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:40.245021 18987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:40.245440 18987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:40.245500 18987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.245546 18987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:40.245573 18987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.252175 18987 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:41983
I20260808 02:20:40.252238 19163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:41983 every 8 connection(s)
I20260808 02:20:40.252580 18987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:20:40.256011 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 18987
I20260808 02:20:40.256126 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:20:40.256359 29382 external_mini_cluster.cc:1428] Setting key c65b7f39f5b605a8acb16ed6cfd1a050
Aug 08 02:20:40 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:40.264614 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.253432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:42905 (local address 127.28.177.190:41243)
0808 02:20:40.253566 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.253568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.253857 (+ 289us) server_negotiation.cc:407] Connection header received
0808 02:20:40.254453 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.254454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.254469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.254539 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:40.255677 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.256563 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.258361 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.258568 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.258632 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.261190 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:40.261193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:40.261193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:40.261203 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:40.262988 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.263525 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.263527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.263527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.263587 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.263878 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.263879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.263880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.264014 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:40.264095 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.264409 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.264515 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 02:20:40.264824 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.253379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:42905)
0808 02:20:40.253658 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.253755 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.254330 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.254665 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.254670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.255320 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:40.255540 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.255544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.256696 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.256698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.258256 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.258261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.258663 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.258664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.258828 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.259434 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:40.259448 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:40.261086 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:40.263140 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:40.263153 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:40.263155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:40.263397 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:40.263669 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:40.263671 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:40.263671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:40.263794 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:40.264100 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:40.264108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:40.264255 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.264370 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.264550 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":125,"threads_started":1}
I20260808 02:20:40.265388 19164 heartbeater.cc:347] Connected to a master server at 127.28.177.190:41243
I20260808 02:20:40.265517 19164 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:40.265756 19164 heartbeater.cc:511] Master 127.28.177.190:41243 requested a full tablet report, sending...
I20260808 02:20:40.266270 18704 ts_manager.cc:194] Registered new tserver with Master: 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
I20260808 02:20:40.266665 18704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:42905
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:20:40.289485 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:41243
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:20:40.386705 19171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:20:40.386943 19171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:20:40.386971 19171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:20:40.388242 19171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:20:40.388293 19171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:20:40.388320 19171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:20:40.389968 19171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:20:40.390327 19171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:20:40.390590 19171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:20:40.396507 19177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:40.396488 19178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:20:40.396492 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:20:40.396513 19171 server_base.cc:1034] running on GCE node
I20260808 02:20:40.396973 19171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:20:40.397362 19171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:20:40.398566 19171 hybrid_clock.cc:648] HybridClock initialized: now 1786155640398548 us; error 48 us; skew 500 ppm
Aug 08 02:20:40 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:20:40.401889 19171 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:20:40.402387 19171 webserver.cc:460] Webserver started at http://127.28.177.131:41649/ using document root <none> and password file <none>
I20260808 02:20:40.402628 19171 fs_manager.cc:359] Metadata directory not provided
I20260808 02:20:40.402676 19171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:20:40.402796 19171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:20:40.403986 19171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "dada577d1cbd46908d4bce911bafd8fb"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "f3b23238a8fd45496073f9256d4a4b45"
server_key_iv: "3d537b1968e4a4bc406e6de465d81b4e"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.404323 19171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "dada577d1cbd46908d4bce911bafd8fb"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "f3b23238a8fd45496073f9256d4a4b45"
server_key_iv: "3d537b1968e4a4bc406e6de465d81b4e"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.406242 19171 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260808 02:20:40.407220 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.407433 19171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:20:40.407527 19171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "dada577d1cbd46908d4bce911bafd8fb"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "f3b23238a8fd45496073f9256d4a4b45"
server_key_iv: "3d537b1968e4a4bc406e6de465d81b4e"
server_key_version: "encryptionkey@0"
I20260808 02:20:40.407589 19171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:20:40.421789 19171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:20:40.424538 19171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:20:40.424665 19171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:20:40.424968 19171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:20:40.425320 19171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:20:40.425359 19171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.425419 19171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:20:40.425452 19171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:20:40.431962 19171 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:32853
I20260808 02:20:40.432149 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:32853 every 8 connection(s)
I20260808 02:20:40.432420 19171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:20:40.436645 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 19171
I20260808 02:20:40.436739 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:20:40.436962 29382 external_mini_cluster.cc:1428] Setting key d998181282d76f634a59d30f4760616f
Aug 08 02:20:40 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:20:40.444046 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.433257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:37367 (local address 127.28.177.190:41243)
0808 02:20:40.433350 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.433353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.433586 (+ 233us) server_negotiation.cc:407] Connection header received
0808 02:20:40.434332 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.434333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.434348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.434422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:40.435467 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.435993 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.437362 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.437665 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.437763 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.440681 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:40.440683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:40.440684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:40.440692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:40.442435 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.442932 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.442933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.442933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.442989 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.443302 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.443304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.443304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.443492 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:40.443567 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.443864 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.443939 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:20:40.444358 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.433223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:37367)
0808 02:20:40.433430 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.433481 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.434197 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.434498 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.434503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.435131 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:20:40.435338 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.435342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.436117 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.436118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.437285 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.437289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.437798 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.437799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.437940 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.438603 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:20:40.438613 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:20:40.440494 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:20:40.442547 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:40.442556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:40.442558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:40.442851 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:40.443074 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:20:40.443075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:20:40.443076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:20:40.443197 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:20:40.443578 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:20:40.443587 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:20:40.443728 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.443830 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.443978 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260808 02:20:40.445140 19351 heartbeater.cc:347] Connected to a master server at 127.28.177.190:41243
I20260808 02:20:40.445240 19351 heartbeater.cc:464] Registering TS with master...
I20260808 02:20:40.445470 19351 heartbeater.cc:511] Master 127.28.177.190:41243 requested a full tablet report, sending...
I20260808 02:20:40.446007 18704 ts_manager.cc:194] Registered new tserver with Master: dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853)
I20260808 02:20:40.446453 18704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:37367
I20260808 02:20:40.449015 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:20:40.456718 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.450387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.28.177.190:41243)
0808 02:20:40.450453 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.450455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.450585 (+ 130us) server_negotiation.cc:407] Connection header received
0808 02:20:40.450688 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.450689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.450700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.450748 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:20:40.451378 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.451868 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.452618 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.452828 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.452934 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.454109 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:20:40.454111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:20:40.454111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:20:40.454119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:20:40.455449 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.455825 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.455827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.455827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.455896 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:20:40.456108 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:20:40.456108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:20:40.456109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:20:40.456282 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:20:40.456393 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.456517 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.456600 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:20:40.458515 18704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:20:40.460371 18704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:20:40.467916 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.463796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.28.177.130:41983)
0808 02:20:40.464077 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.464082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.464095 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:20:40.464282 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.464285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.464303 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.464389 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:40.464566 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.465694 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.466895 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.467426 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.467539 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.467617 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.467767 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.467817 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260808 02:20:40.467903 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.463402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:41983 (local address 127.0.0.1:35574)
0808 02:20:40.463926 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.463937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.464169 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.464407 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.464408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.464411 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:40.464472 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.464474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.465808 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.465809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.466791 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.466795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.467539 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.467540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.467633 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.467651 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.467749 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.467814 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":50,"thread_start_us":72,"threads_started":1}
I20260808 02:20:40.468238 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.463202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:33409 (local address 127.0.0.1:53116)
0808 02:20:40.463459 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.463472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.463531 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.464293 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.464294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.464297 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:40.464355 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.464357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.465794 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.465796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.466992 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.466995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.467955 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.467956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.468035 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.468040 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.468080 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.468135 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":35,"thread_start_us":96,"threads_started":1}
I20260808 02:20:40.468237 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.463124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:32853 (local address 127.0.0.1:53420)
0808 02:20:40.463347 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.463389 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.463456 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.463897 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.463901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.463904 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:40.464006 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.464008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.465306 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.465307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.466816 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.466819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.467997 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.467998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.468064 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.468069 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.468122 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.468159 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260808 02:20:40.468451 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.463251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.28.177.131:32853)
0808 02:20:40.463622 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.463628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.463640 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:20:40.463686 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.463688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.463706 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.463795 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:40.464107 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.465165 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.466956 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.467741 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.467831 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.467960 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.468289 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.468344 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":110,"threads_started":1}
I20260808 02:20:40.468739 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.463381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.28.177.129:33409)
0808 02:20:40.463656 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.463660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.463668 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:20:40.463715 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.463717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.463734 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.464440 (+ 706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:40.464530 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.465663 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.467099 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.467826 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.468421 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.468508 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.468541 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.468594 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260808 02:20:40.469563 19251 tablet_service.cc:1467] Processing CreateTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 (DEFAULT_TABLE table=test-table [id=ce5c562349ff43ddb00f91a6dc8696ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:40.469604 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 (DEFAULT_TABLE table=test-table [id=ce5c562349ff43ddb00f91a6dc8696ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:40.469718 19066 tablet_service.cc:1467] Processing CreateTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 (DEFAULT_TABLE table=test-table [id=ce5c562349ff43ddb00f91a6dc8696ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:20:40.470041 19251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082f44cfca5744fe8d20f0d7f6141927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:40.470031 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082f44cfca5744fe8d20f0d7f6141927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:40.470052 19066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082f44cfca5744fe8d20f0d7f6141927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:20:40.473434 19378 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Bootstrap starting.
I20260808 02:20:40.473989 19381 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Bootstrap starting.
I20260808 02:20:40.474352 19378 tablet_bootstrap.cc:654] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:40.474666 19381 tablet_bootstrap.cc:654] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:40.474982 19378 log.cc:824] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:40.475317 19381 log.cc:824] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:40.475731 19379 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap starting.
I20260808 02:20:40.476004 19378 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: No bootstrap required, opened a new log
I20260808 02:20:40.476073 19378 ts_tablet_manager.cc:1397] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:20:40.476193 19381 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: No bootstrap required, opened a new log
I20260808 02:20:40.476287 19381 ts_tablet_manager.cc:1397] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:20:40.477828 19381 raft_consensus.cc:348] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.478029 19381 raft_consensus.cc:374] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:40.478080 19381 raft_consensus.cc:729] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7004854aedc24e9e93065d0ee39fb6b3, State: Initialized, Role: FOLLOWER
I20260808 02:20:40.478232 19381 consensus_queue.cc:260] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.478493 19378 raft_consensus.cc:348] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.478703 19378 raft_consensus.cc:374] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:40.478756 19378 raft_consensus.cc:729] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dada577d1cbd46908d4bce911bafd8fb, State: Initialized, Role: FOLLOWER
I20260808 02:20:40.478927 19381 ts_tablet_manager.cc:1428] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:20:40.478888 19378 consensus_queue.cc:260] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.478978 19164 heartbeater.cc:503] Master 127.28.177.190:41243 was elected leader, sending a full tablet report...
I20260808 02:20:40.479206 19378 ts_tablet_manager.cc:1428] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:20:40.479228 19351 heartbeater.cc:503] Master 127.28.177.190:41243 was elected leader, sending a full tablet report...
I20260808 02:20:40.479787 19379 tablet_bootstrap.cc:654] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Neither blocks nor log segments found. Creating new log.
I20260808 02:20:40.480458 19379 log.cc:824] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Log is configured to *not* fsync() on all Append() calls
I20260808 02:20:40.481395 19379 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: No bootstrap required, opened a new log
I20260808 02:20:40.481463 19379 ts_tablet_manager.cc:1397] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 02:20:40.482929 19379 raft_consensus.cc:348] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.483093 19379 raft_consensus.cc:374] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:20:40.483120 19379 raft_consensus.cc:729] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b749f373d04a7ca94fbe58035ed119, State: Initialized, Role: FOLLOWER
I20260808 02:20:40.483209 19379 consensus_queue.cc:260] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.483434 19379 ts_tablet_manager.cc:1428] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:20:40.483542 18980 heartbeater.cc:503] Master 127.28.177.190:41243 was elected leader, sending a full tablet report...
W20260808 02:20:40.503275 19165 tablet.cc:2369] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:20:40.562623 19385 raft_consensus.cc:482] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:20:40.562773 19385 raft_consensus.cc:504] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.563244 19385 leader_election.cc:288] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409), 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
I20260808 02:20:40.566890 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.563819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:36643 (local address 127.28.177.130:41983)
0808 02:20:40.563902 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.563905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.563937 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:20:40.563966 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.563967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.563976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.564180 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:40.564389 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.564908 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.566071 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.566463 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.566604 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.566616 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.566759 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.566801 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:20:40.566915 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.563474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:33409 (local address 127.28.177.131:48393)
0808 02:20:40.563591 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.563609 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.563683 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.563975 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.563976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.563979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:40.564065 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.564068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.564963 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.564965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.565835 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.565838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.566571 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.566573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.566672 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.566676 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.566744 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.566793 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":21}
I20260808 02:20:40.566926 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.563597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:41983 (local address 127.28.177.131:36643)
0808 02:20:40.563833 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.563845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:20:40.563904 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:20:40.564223 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:20:40.564224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:20:40.564227 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:20:40.564301 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.564303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.565159 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.565160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.565966 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:20:40.565969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.566615 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.566616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:20:40.566674 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.566677 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:20:40.566749 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:20:40.566794 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":170,"threads_started":1}
I20260808 02:20:40.566907 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.563640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:48393 (local address 127.28.177.129:33409)
0808 02:20:40.563729 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.563731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.563743 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:20:40.563780 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.563781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.563793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.563891 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:20:40.564214 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.564792 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.565936 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.566442 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.566505 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.566514 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.566768 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.566814 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:20:40.567351 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082f44cfca5744fe8d20f0d7f6141927" candidate_uuid: "dada577d1cbd46908d4bce911bafd8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7004854aedc24e9e93065d0ee39fb6b3" is_pre_election: true
I20260808 02:20:40.567394 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082f44cfca5744fe8d20f0d7f6141927" candidate_uuid: "dada577d1cbd46908d4bce911bafd8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b749f373d04a7ca94fbe58035ed119" is_pre_election: true
I20260808 02:20:40.567553 19095 raft_consensus.cc:2454] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dada577d1cbd46908d4bce911bafd8fb in term 0.
I20260808 02:20:40.567579 18913 raft_consensus.cc:2454] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dada577d1cbd46908d4bce911bafd8fb in term 0.
I20260808 02:20:40.567832 19192 leader_election.cc:302] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb; no voters:
I20260808 02:20:40.567991 19385 raft_consensus.cc:2781] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:20:40.568092 19385 raft_consensus.cc:482] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:20:40.568122 19385 raft_consensus.cc:3037] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:40.569001 19385 raft_consensus.cc:504] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.569147 19385 leader_election.cc:288] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [CANDIDATE]: Term 1 election: Requested vote from peers 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409), 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
I20260808 02:20:40.569336 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082f44cfca5744fe8d20f0d7f6141927" candidate_uuid: "dada577d1cbd46908d4bce911bafd8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b749f373d04a7ca94fbe58035ed119"
I20260808 02:20:40.569396 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082f44cfca5744fe8d20f0d7f6141927" candidate_uuid: "dada577d1cbd46908d4bce911bafd8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7004854aedc24e9e93065d0ee39fb6b3"
I20260808 02:20:40.569422 18913 raft_consensus.cc:3037] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:40.569511 19095 raft_consensus.cc:3037] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:20:40.570541 18913 raft_consensus.cc:2454] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dada577d1cbd46908d4bce911bafd8fb in term 1.
I20260808 02:20:40.570741 19095 raft_consensus.cc:2454] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dada577d1cbd46908d4bce911bafd8fb in term 1.
I20260808 02:20:40.570874 19192 leader_election.cc:302] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b749f373d04a7ca94fbe58035ed119, dada577d1cbd46908d4bce911bafd8fb; no voters:
I20260808 02:20:40.571180 19385 raft_consensus.cc:2781] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:20:40.571403 19385 raft_consensus.cc:686] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 1 LEADER]: Becoming Leader. State: Replica: dada577d1cbd46908d4bce911bafd8fb, State: Running, Role: LEADER
I20260808 02:20:40.571527 19385 consensus_queue.cc:237] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:20:40.572666 18704 catalog_manager.cc:5495] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb reported cstate change: term changed from 0 to 1, leader changed from <none> to dada577d1cbd46908d4bce911bafd8fb (127.28.177.131). New cstate: current_term: 1 leader_uuid: "dada577d1cbd46908d4bce911bafd8fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } health_report { overall_health: UNKNOWN } } }
W20260808 02:20:40.576606 18981 tablet.cc:2369] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:20:40.586419 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.583556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.28.177.131:32853)
0808 02:20:40.583638 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 02:20:40.583640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:20:40.583652 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:20:40.583702 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:20:40.583703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:20:40.583728 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:20:40.583828 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:20:40.583966 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.584568 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.585473 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:20:40.585713 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:20:40.585797 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:20:40.585998 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:20:40.586125 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:20:40.586230 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:20:40.586273 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 02:20:40.586299 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:20:40.588690 19095 raft_consensus.cc:1261] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Refusing update from remote peer dada577d1cbd46908d4bce911bafd8fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:40.588725 18913 raft_consensus.cc:1261] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Refusing update from remote peer dada577d1cbd46908d4bce911bafd8fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:20:40.589116 19385 consensus_queue.cc:1035] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:40.589337 19390 consensus_queue.cc:1035] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:20:40.592132 19395 mvcc.cc:205] Tried to move back new op lower bound from 7316093503848116224 to 7316093503781978112. Current Snapshot: MvccSnapshot[applied={T|T < 7316093503848116224}]
I20260808 02:20:40.592427 19396 mvcc.cc:205] Tried to move back new op lower bound from 7316093503848116224 to 7316093503781978112. Current Snapshot: MvccSnapshot[applied={T|T < 7316093503848116224}]
I20260808 02:20:40.594970 18704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:40.595088 18704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:20:40.595971 18704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Sending DeleteTablet for 3 replicas of tablet 082f44cfca5744fe8d20f0d7f6141927
I20260808 02:20:40.596297 19251 tablet_service.cc:1514] Processing DeleteTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:40 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:53420
I20260808 02:20:40.596342 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 18795
I20260808 02:20:40.596413 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:40 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:53116
I20260808 02:20:40.601408 19066 tablet_service.cc:1514] Processing DeleteTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:40 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:35574
I20260808 02:20:40.606295 19409 tablet_replica.cc:330] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: stopping tablet replica
I20260808 02:20:40.606501 19409 raft_consensus.cc:2229] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:20:40.606696 19409 raft_consensus.cc:2258] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:40.606961 19408 tablet_replica.cc:330] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: stopping tablet replica
I20260808 02:20:40.607125 19408 raft_consensus.cc:2229] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:20:40.607331 19408 raft_consensus.cc:2258] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:20:40.608687 19409 ts_tablet_manager.cc:1905] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:40.611347 19409 ts_tablet_manager.cc:1918] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:40.611451 19409 log.cc:1196] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/082f44cfca5744fe8d20f0d7f6141927
I20260808 02:20:40.611886 19409 ts_tablet_manager.cc:1939] T 082f44cfca5744fe8d20f0d7f6141927 P 7004854aedc24e9e93065d0ee39fb6b3: Deleting consensus metadata
I20260808 02:20:40.612584 18681 catalog_manager.cc:4841] TS 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983): tablet 082f44cfca5744fe8d20f0d7f6141927 (table test-table [id=ce5c562349ff43ddb00f91a6dc8696ee]) successfully deleted
I20260808 02:20:40.614136 19408 ts_tablet_manager.cc:1905] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:20:40.616688 19408 ts_tablet_manager.cc:1918] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:20:40.616794 19408 log.cc:1196] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/082f44cfca5744fe8d20f0d7f6141927
I20260808 02:20:40.617491 19408 ts_tablet_manager.cc:1939] T 082f44cfca5744fe8d20f0d7f6141927 P dada577d1cbd46908d4bce911bafd8fb: Deleting consensus metadata
I20260808 02:20:40.619112 18678 catalog_manager.cc:4841] TS dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853): tablet 082f44cfca5744fe8d20f0d7f6141927 (table test-table [id=ce5c562349ff43ddb00f91a6dc8696ee]) successfully deleted
W20260808 02:20:40.628952 18681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:33409 (error 108)
I20260808 02:20:40.629091 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 18987
I20260808 02:20:40.632025 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:20:40.631829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:33409 (local address 127.0.0.1:53120)
0808 02:20:40.631916 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0808 02:20:40.631968 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:20:40.632207 18681 catalog_manager.cc:4584] TS 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 082f44cfca5744fe8d20f0d7f6141927: Network error: Client connection negotiation failed: client connection to 127.28.177.129:33409: connect: Connection refused (error 111)
I20260808 02:20:40.645396 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 19171
I20260808 02:20:40.663439 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 18653
I20260808 02:21:00.678849 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41243
--webserver_interface=127.28.177.190
--webserver_port=33287
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:00.766219 19414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:00.766444 19414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:00.766469 19414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:00.767720 19414 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:21:00.767766 19414 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:21:00.767779 19414 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:21:00.767791 19414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:00.767804 19414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:00.767819 19414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:00.767831 19414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:00.767874 19414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:00.769511 19414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:00.769841 19414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:00.770095 19414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:00.781677 19429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:00.781790 19425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:00.781706 19414 server_base.cc:1034] running on GCE node
W20260808 02:21:00.781664 19427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:00.782298 19414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:00.782655 19414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:00.783841 19414 hybrid_clock.cc:648] HybridClock initialized: now 1786155660783813 us; error 41 us; skew 500 ppm
Aug 08 02:21:00 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:00.787151 19414 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:21:00.787748 19414 webserver.cc:460] Webserver started at http://127.28.177.190:33287/ using document root <none> and password file <none>
I20260808 02:21:00.788120 19414 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:00.788236 19414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:00.789997 19414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:00.790884 19437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:00.791100 19414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:00.791234 19414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "d6c399d2fc51444bb3a30a5dddca2799"
format_stamp: "Formatted at 2026-08-08 02:20:39 on dist-test-slave-3h5h"
server_key: "065efddd121fb6443974527231466a4f"
server_key_iv: "cb19085afcf224426143b56ba39c684a"
server_key_version: "encryptionkey@0"
I20260808 02:21:00.791738 19414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:00.808313 19414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:00.810600 19414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:00.810796 19414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:00.815732 19414 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:41243
I20260808 02:21:00.815776 19508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:41243 every 8 connection(s)
I20260808 02:21:00.816272 19414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:00.818008 19510 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:21:00.818709 19510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap starting.
I20260808 02:21:00.826560 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 19414
I20260808 02:21:00.826999 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:33409
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:41243
--webserver_port=36679
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:21:00.829522 19510 log.cc:824] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:00.831636 19510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:21:00.832008 19510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap complete.
I20260808 02:21:00.834610 19510 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:00.834919 19510 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c399d2fc51444bb3a30a5dddca2799, State: Initialized, Role: FOLLOWER
I20260808 02:21:00.835142 19510 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:00.835246 19510 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:00.835311 19510 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:00.835362 19510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:21:00.836462 19510 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:00.836593 19510 leader_election.cc:302] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c399d2fc51444bb3a30a5dddca2799; no voters:
I20260808 02:21:00.836818 19510 leader_election.cc:288] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:21:00.836884 19517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:21:00.837014 19517 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 LEADER]: Becoming Leader. State: Replica: d6c399d2fc51444bb3a30a5dddca2799, State: Running, Role: LEADER
I20260808 02:21:00.837150 19510 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:00.837188 19517 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:00.837671 19518 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6c399d2fc51444bb3a30a5dddca2799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } } }
I20260808 02:21:00.837800 19518 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:00.838065 19519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c399d2fc51444bb3a30a5dddca2799. Latest consensus state: current_term: 2 leader_uuid: "d6c399d2fc51444bb3a30a5dddca2799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } } }
I20260808 02:21:00.838176 19519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:00.838482 19532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:00.839767 19532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:00.839923 19532 catalog_manager.cc:1259] Loaded cluster ID: 2731e109ee424054abd66c61fdff3725
I20260808 02:21:00.839980 19532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:00.841586 19532 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:00.841804 19532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Loaded TSK: 0
I20260808 02:21:00.856658 19532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Generated new TSK 1
I20260808 02:21:00.856810 19532 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:21:00.928812 19515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:00.929119 19515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:00.929145 19515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:00.930387 19515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:00.930431 19515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:00.930460 19515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:00.931986 19515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:33409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=36679
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:00.932416 19515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:00.932678 19515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:00.938215 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:00.938235 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:00.938395 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:00.938683 19515 server_base.cc:1034] running on GCE node
I20260808 02:21:00.938956 19515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:00.939272 19515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:00.940442 19515 hybrid_clock.cc:648] HybridClock initialized: now 1786155660940432 us; error 30 us; skew 500 ppm
Aug 08 02:21:00 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:00.943483 19515 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:21:00.944080 19515 webserver.cc:460] Webserver started at http://127.28.177.129:36679/ using document root <none> and password file <none>
I20260808 02:21:00.944346 19515 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:00.944433 19515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:00.946058 19515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:00.946884 19568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:00.947093 19515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:00.947286 19515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "28b749f373d04a7ca94fbe58035ed119"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "61f971c4647110d527e705f31f3f1c50"
server_key_iv: "752c4e05ba34d7fc5fb1833aee7455eb"
server_key_version: "encryptionkey@0"
I20260808 02:21:00.947633 19515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:00.959344 19515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:00.962190 19515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:00.962387 19515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:00.962714 19515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:00.963316 19576 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:21:00.965169 19515 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:21:00.965233 19515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 02:21:00.965262 19515 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:21:00.966034 19515 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:21:00.966086 19515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:00.966156 19576 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap starting.
I20260808 02:21:00.975406 19515 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:33409
I20260808 02:21:00.975988 19515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:00.977106 19725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:33409 every 8 connection(s)
I20260808 02:21:00.980414 19576 log.cc:824] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:00.983728 19576 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:21:00.984117 19576 tablet_bootstrap.cc:492] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap complete.
I20260808 02:21:00.984546 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 19515
I20260808 02:21:00.984970 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:41983
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:41243
--webserver_port=33987
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:21:00.985749 19576 ts_tablet_manager.cc:1397] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20260808 02:21:00.987342 19576 raft_consensus.cc:348] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:21:00.987654 19576 raft_consensus.cc:729] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b749f373d04a7ca94fbe58035ed119, State: Initialized, Role: FOLLOWER
I20260808 02:21:00.987843 19576 consensus_queue.cc:260] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } }
I20260808 02:21:00.988206 19576 ts_tablet_manager.cc:1428] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Aug 08 02:21:00 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:00.996459 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:00.977979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:51583 (local address 127.28.177.190:41243)
0808 02:21:00.980762 (+ 2783us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:00.980775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:00.981029 (+ 254us) server_negotiation.cc:407] Connection header received
0808 02:21:00.982675 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:00.982684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:00.982757 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:00.982930 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:00.983964 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:00.985623 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:00.987270 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:00.987944 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:00.988143 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:00.991474 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:00.991479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:00.991480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:00.991494 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:00.993936 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:00.994649 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:00.994654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:00.994654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:00.994792 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:00.995169 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:00.995170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:00.995171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:00.995554 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:00.995692 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:00.996030 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 02:21:00.996287 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2708,"thread_start_us":112,"threads_started":1}
I20260808 02:21:00.996457 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:00.977891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:51583)
0808 02:21:00.980789 (+ 2898us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:00.980868 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:00.982385 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:00.982986 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:00.982992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:00.983453 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:00.983795 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:00.983804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:00.985768 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:00.985770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:00.987162 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:00.987165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:00.988241 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:00.988243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:00.988403 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:00.989158 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:00.989169 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:00.991333 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:00.994065 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:00.994078 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:00.994081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:00.994520 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:00.994930 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:00.994931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:00.994931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:00.995080 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:00.995661 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:00.995672 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:00.995832 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 02:21:00.995962 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 02:21:00.996195 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2837,"thread_start_us":118,"threads_started":1}
I20260808 02:21:00.997448 19726 heartbeater.cc:347] Connected to a master server at 127.28.177.190:41243
I20260808 02:21:00.997586 19726 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:00.997805 19726 heartbeater.cc:511] Master 127.28.177.190:41243 requested a full tablet report, sending...
I20260808 02:21:00.998760 19462 ts_manager.cc:194] Registered new tserver with Master: 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409)
I20260808 02:21:01.000005 19462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:51583
I20260808 02:21:01.001241 19726 heartbeater.cc:503] Master 127.28.177.190:41243 was elected leader, sending a full tablet report...
Aug 08 02:21:01 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.129@KRBTEST.COM
I20260808 02:21:01.009639 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:00.999504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:33409 (local address 127.0.0.1:48928)
0808 02:21:00.999725 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:00.999769 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:00.999843 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.000098 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.000112 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.000459 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:01.000518 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.000520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.001663 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.001665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.002861 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.002864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.003370 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.003372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.003461 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.004191 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:01.004198 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:01.005948 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:01.008288 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:01.008296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:01.008298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:01.008545 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:01.008802 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:01.008803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:01.008803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:01.008928 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:01.009316 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:01.009323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:01.009372 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.009443 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.009552 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":127,"threads_started":1}
I20260808 02:21:01.009895 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:00.999580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.28.177.129:33409)
0808 02:21:00.999798 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:00.999804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:00.999853 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:21:00.999950 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:00.999952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:00.999972 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.000069 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:01.000625 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.001547 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.002955 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.003280 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.003350 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.006042 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:01.006053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:01.006053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:01.006061 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:01.008167 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.008644 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.008648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.008648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.008725 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.008997 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.008998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.008998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.009239 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:01.009333 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.009515 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.009633 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260808 02:21:01.010334 19637 tablet_service.cc:1514] Processing DeleteTablet for tablet 082f44cfca5744fe8d20f0d7f6141927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:20:40 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:48928
I20260808 02:21:01.010910 19742 tablet_replica.cc:330] stopping tablet replica
I20260808 02:21:01.011034 19742 raft_consensus.cc:2229] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:01.011112 19742 raft_consensus.cc:2258] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:01.011541 19742 ts_tablet_manager.cc:1905] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:01.014031 19742 ts_tablet_manager.cc:1918] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:01.014098 19742 log.cc:1196] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal/wals/082f44cfca5744fe8d20f0d7f6141927
I20260808 02:21:01.014382 19742 ts_tablet_manager.cc:1939] T 082f44cfca5744fe8d20f0d7f6141927 P 28b749f373d04a7ca94fbe58035ed119: Deleting consensus metadata
I20260808 02:21:01.014983 19443 catalog_manager.cc:4841] TS 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409): tablet 082f44cfca5744fe8d20f0d7f6141927 (table test-table [id=ce5c562349ff43ddb00f91a6dc8696ee]) successfully deleted
W20260808 02:21:01.082501 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:01.082755 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:01.082800 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:01.084012 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:01.084085 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:01.084131 19735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:01.085716 19735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:41983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=33987
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:01.086026 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:01.086284 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:01.092231 19748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:01.092214 19755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:01.092193 19750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:01.092722 19735 server_base.cc:1034] running on GCE node
I20260808 02:21:01.093005 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:01.093250 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:01.094410 19735 hybrid_clock.cc:648] HybridClock initialized: now 1786155661094382 us; error 42 us; skew 500 ppm
Aug 08 02:21:01 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:01.097689 19735 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:21:01.098229 19735 webserver.cc:460] Webserver started at http://127.28.177.130:33987/ using document root <none> and password file <none>
I20260808 02:21:01.098510 19735 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:01.098613 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:01.100126 19735 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:01.101100 19768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:01.101336 19735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:01.101454 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "7004854aedc24e9e93065d0ee39fb6b3"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "ec715513df9c2f82869b44fce5fb8a7a"
server_key_iv: "74c97776436ec2bfef108b6ff17fce34"
server_key_version: "encryptionkey@0"
I20260808 02:21:01.101857 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:01.114187 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:01.117396 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:01.117583 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:01.117849 19735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:01.118204 19735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:01.118263 19735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:01.118309 19735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:01.118333 19735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:01.124891 19735 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:41983
I20260808 02:21:01.124924 19942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:41983 every 8 connection(s)
I20260808 02:21:01.125308 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:21:01 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:01.134829 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 19735
I20260808 02:21:01.135556 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:32853
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:41243
--webserver_port=41649
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:21:01.136646 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.126079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:44015 (local address 127.28.177.190:41243)
0808 02:21:01.126130 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.126133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.126397 (+ 264us) server_negotiation.cc:407] Connection header received
0808 02:21:01.127086 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.127087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.127099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.127178 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:01.128244 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.128772 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.129846 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.130072 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.130155 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.132492 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:01.132494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:01.132495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:01.132503 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:01.134473 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.135160 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.135161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.135162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.135289 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.135756 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.135757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.135758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.135918 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:01.136029 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.136394 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.136548 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:21:01.137032 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.126050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:44015)
0808 02:21:01.126255 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.126303 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:01.126868 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.127275 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.127279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.127909 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:01.128124 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.128130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.128896 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.128897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.129741 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.129744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.130166 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.130167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.130301 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.130881 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:01.130890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:01.132376 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:01.134652 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:01.134662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:01.134664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:01.134995 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:01.135434 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:01.135435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:01.135435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:01.135595 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:01.136018 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:01.136026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:01.136232 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.136357 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.136610 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20260808 02:21:01.137761 19943 heartbeater.cc:347] Connected to a master server at 127.28.177.190:41243
I20260808 02:21:01.137856 19943 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:01.138213 19943 heartbeater.cc:511] Master 127.28.177.190:41243 requested a full tablet report, sending...
I20260808 02:21:01.138722 19462 ts_manager.cc:194] Registered new tserver with Master: 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
I20260808 02:21:01.139124 19462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:44015
W20260808 02:21:01.230764 19947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:01.231015 19947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:01.231045 19947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:01.232266 19947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:01.232316 19947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:01.232347 19947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:01.233978 19947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:32853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=41649
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:01.234251 19947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:01.234543 19947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:21:01.240741 19947 server_base.cc:1034] running on GCE node
W20260808 02:21:01.240664 19953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:01.240701 19958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:01.240684 19960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:01.241196 19947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:01.241542 19947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:01.242735 19947 hybrid_clock.cc:648] HybridClock initialized: now 1786155661242724 us; error 64 us; skew 500 ppm
Aug 08 02:21:01 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:01.245604 19947 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:21:01.246134 19947 webserver.cc:460] Webserver started at http://127.28.177.131:41649/ using document root <none> and password file <none>
I20260808 02:21:01.246387 19947 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:01.246488 19947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:01.247972 19947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:01.248728 19971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:01.249179 19947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:01.249328 19947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "dada577d1cbd46908d4bce911bafd8fb"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "f3b23238a8fd45496073f9256d4a4b45"
server_key_iv: "3d537b1968e4a4bc406e6de465d81b4e"
server_key_version: "encryptionkey@0"
I20260808 02:21:01.249786 19947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:01.279342 19947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:01.281816 19947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:01.282007 19947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:01.282399 19947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:01.282819 19947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:01.282888 19947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:01.282928 19947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:01.282958 19947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:01.289744 19947 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:32853
I20260808 02:21:01.289803 20133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:32853 every 8 connection(s)
I20260808 02:21:01.290169 19947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:01.294090 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 19947
Aug 08 02:21:01 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:01.301702 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.290929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:42809 (local address 127.28.177.190:41243)
0808 02:21:01.291004 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.291007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.291349 (+ 342us) server_negotiation.cc:407] Connection header received
0808 02:21:01.291870 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.291872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.291883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.291940 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:01.292954 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.293478 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.294921 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.295270 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.295349 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.298134 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:01.298136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:01.298136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:01.298145 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:01.299728 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.300331 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.300333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.300333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.300432 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.300968 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.300970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.300970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.301116 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:01.301202 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.301504 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.301599 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:21:01.301916 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.290969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:42809)
0808 02:21:01.291166 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.291236 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:01.291771 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.291996 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.291999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.292621 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:01.292850 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.292855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.293574 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.293574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.294838 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.294841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.295355 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.295356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.295468 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.296094 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:01.296106 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:01.298023 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:01.299825 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:01.299835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:01.299837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:01.300247 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:01.300580 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:01.300581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:01.300581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:01.300809 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:01.301192 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:01.301201 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:01.301352 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.301466 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.301661 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260808 02:21:01.303411 20137 heartbeater.cc:347] Connected to a master server at 127.28.177.190:41243
I20260808 02:21:01.303546 20137 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:01.303782 20137 heartbeater.cc:511] Master 127.28.177.190:41243 requested a full tablet report, sending...
I20260808 02:21:01.304214 19462 ts_manager.cc:194] Registered new tserver with Master: dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853)
I20260808 02:21:01.304561 19462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:42809
I20260808 02:21:01.306402 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.294542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.28.177.190:41243)
0808 02:21:01.294715 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.294717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.294800 (+ 83us) server_negotiation.cc:407] Connection header received
0808 02:21:01.294865 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.294866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.294880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.294954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:01.296934 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.297709 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.299237 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.299784 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.299879 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.302933 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:01.302934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:01.302934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:01.302941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:01.305186 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.305569 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.305570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.305571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.305636 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.305848 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.305849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.305849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.305998 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:01.306166 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.306207 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.306294 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260808 02:21:01.307384 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:01.310420 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.307792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.28.177.190:41243)
0808 02:21:01.307845 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.307847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.307948 (+ 101us) server_negotiation.cc:407] Connection header received
0808 02:21:01.307995 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.307996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.308026 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.308101 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:01.308252 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.308803 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.309667 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.309879 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.309944 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.310089 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:01.310137 (+ 48us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 02:21:01.310370 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56636: token expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260808 02:21:01.310521 20144 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56636: token expired
W20260808 02:21:01.310519 19359 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.177.190:41243)
I20260808 02:21:01.316962 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.310845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.28.177.190:41243)
0808 02:21:01.310885 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.310886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.310894 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:21:01.310912 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.310913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.310921 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.310986 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:01.311463 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.311953 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.312744 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.313003 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.313061 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.314184 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:01.314186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:01.314186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:01.314193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:01.315815 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.316191 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.316192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.316193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.316256 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:01.316456 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:01.316456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:01.316457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:01.316568 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:01.316665 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.316734 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.316803 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:21:01.318305 19462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:01.320120 19462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:01.323020 19637 tablet_service.cc:1467] Processing CreateTablet for tablet d32942ab3a904f18864c9c294b981194 (DEFAULT_TABLE table=test-table [id=1ec815e4ad8b4073b523bad2b6231a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:01.323254 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32942ab3a904f18864c9c294b981194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:01.325074 19576 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap starting.
I20260808 02:21:01.325505 19576 tablet_bootstrap.cc:654] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:01.326241 19576 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: No bootstrap required, opened a new log
I20260808 02:21:01.326296 19576 ts_tablet_manager.cc:1397] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:01.326669 19576 raft_consensus.cc:348] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.326753 19576 raft_consensus.cc:374] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:01.326809 19576 raft_consensus.cc:729] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b749f373d04a7ca94fbe58035ed119, State: Initialized, Role: FOLLOWER
I20260808 02:21:01.326862 19576 consensus_queue.cc:260] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.326958 19576 ts_tablet_manager.cc:1428] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:01.327277 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.322512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.28.177.130:41983)
0808 02:21:01.322712 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.322718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.322769 (+ 51us) server_negotiation.cc:407] Connection header received
0808 02:21:01.322856 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.322858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.322875 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.322956 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:01.323250 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.324317 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.325829 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.326487 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.326584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.326652 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.327122 (+ 470us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.327179 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260808 02:21:01.327513 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.322419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:41983 (local address 127.0.0.1:40394)
0808 02:21:01.322714 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.322725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:01.322781 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.323005 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.323006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.323009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:01.323086 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.323088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.324419 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.324421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.325712 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.325715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.326597 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.326599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.326681 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.326704 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.326918 (+ 214us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.327428 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260808 02:21:01.328336 19839 tablet_service.cc:1467] Processing CreateTablet for tablet d32942ab3a904f18864c9c294b981194 (DEFAULT_TABLE table=test-table [id=1ec815e4ad8b4073b523bad2b6231a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:01.328758 19839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32942ab3a904f18864c9c294b981194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:01.329217 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.323448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.28.177.131:32853)
0808 02:21:01.325365 (+ 1917us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.325371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.325383 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:21:01.325425 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.325427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.325446 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.325700 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:01.325825 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.326899 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.328159 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.328816 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.328950 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.329006 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.329121 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.329154 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1872,"thread_start_us":115,"threads_started":1}
I20260808 02:21:01.329267 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.322320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:32853 (local address 127.0.0.1:42152)
0808 02:21:01.322467 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.322479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:01.322524 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.325705 (+ 3181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.325707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.325720 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:01.325767 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.325768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.327012 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.327014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.327994 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.327997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.328949 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.328950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.329032 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.329037 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.329105 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.329155 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260808 02:21:01.332244 20154 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Bootstrap starting.
I20260808 02:21:01.332815 20043 tablet_service.cc:1467] Processing CreateTablet for tablet d32942ab3a904f18864c9c294b981194 (DEFAULT_TABLE table=test-table [id=1ec815e4ad8b4073b523bad2b6231a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:01.333107 20154 tablet_bootstrap.cc:654] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:01.333218 20043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32942ab3a904f18864c9c294b981194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:01.333832 20154 log.cc:824] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:01.334657 20154 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: No bootstrap required, opened a new log
I20260808 02:21:01.334738 20154 ts_tablet_manager.cc:1397] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:21:01.336990 20154 raft_consensus.cc:348] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.337206 20154 raft_consensus.cc:374] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:01.337267 20154 raft_consensus.cc:729] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7004854aedc24e9e93065d0ee39fb6b3, State: Initialized, Role: FOLLOWER
I20260808 02:21:01.337393 20154 consensus_queue.cc:260] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.337728 20154 ts_tablet_manager.cc:1428] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260808 02:21:01.337785 19943 heartbeater.cc:503] Master 127.28.177.190:41243 was elected leader, sending a full tablet report...
I20260808 02:21:01.337986 20162 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Bootstrap starting.
I20260808 02:21:01.338770 20162 tablet_bootstrap.cc:654] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:01.339345 20162 log.cc:824] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:01.340229 20162 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: No bootstrap required, opened a new log
I20260808 02:21:01.340289 20162 ts_tablet_manager.cc:1397] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:01.341975 20162 raft_consensus.cc:348] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.342127 20162 raft_consensus.cc:374] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:01.342150 20162 raft_consensus.cc:729] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dada577d1cbd46908d4bce911bafd8fb, State: Initialized, Role: FOLLOWER
I20260808 02:21:01.342275 20162 consensus_queue.cc:260] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.342576 20162 ts_tablet_manager.cc:1428] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:21:01.342630 20137 heartbeater.cc:503] Master 127.28.177.190:41243 was elected leader, sending a full tablet report...
W20260808 02:21:01.376044 19945 tablet.cc:2369] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:21:01.476977 19728 tablet.cc:2369] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:21:01.540882 20138 tablet.cc:2369] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:21:01.703500 20163 raft_consensus.cc:482] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:01.703709 20163 raft_consensus.cc:504] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.704275 20163 leader_election.cc:288] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853), 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409)
I20260808 02:21:01.708328 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.704659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:39757 (local address 127.28.177.131:32853)
0808 02:21:01.704716 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.704718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.704732 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:21:01.704778 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.704779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.704794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.704929 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:01.705185 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.706043 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.707533 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.707953 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.708047 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.708058 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.708213 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.708253 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:21:01.708345 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.704496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:32853 (local address 127.28.177.130:39757)
0808 02:21:01.704651 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.704660 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:01.704721 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.704960 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.704962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.704966 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:01.705076 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.705078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.706169 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.706170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.707428 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.707432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.708049 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.708050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.708135 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.708138 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.708208 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.708252 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260808 02:21:01.708686 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.704613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:52223 (local address 127.28.177.129:33409)
0808 02:21:01.704774 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.704776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.704784 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:21:01.704815 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.704816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.704864 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.704958 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:01.705221 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.706048 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.707614 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.708114 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.708223 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.708317 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.708371 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.708597 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260808 02:21:01.708741 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.704498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:33409 (local address 127.28.177.130:52223)
0808 02:21:01.704646 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.704660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:01.704721 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:01.704960 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:01.704962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:01.704964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:01.705075 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.705078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.706190 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.706191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.707505 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:01.707508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.708219 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.708220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:01.708292 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.708297 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:01.708354 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:21:01.708404 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":40,"thread_start_us":60,"threads_started":1}
I20260808 02:21:01.708709 20072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32942ab3a904f18864c9c294b981194" candidate_uuid: "7004854aedc24e9e93065d0ee39fb6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dada577d1cbd46908d4bce911bafd8fb" is_pre_election: true
I20260808 02:21:01.708901 20072 raft_consensus.cc:2454] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7004854aedc24e9e93065d0ee39fb6b3 in term 0.
I20260808 02:21:01.709170 19770 leader_election.cc:302] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb; no voters:
I20260808 02:21:01.709157 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32942ab3a904f18864c9c294b981194" candidate_uuid: "7004854aedc24e9e93065d0ee39fb6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b749f373d04a7ca94fbe58035ed119" is_pre_election: true
I20260808 02:21:01.709309 20163 raft_consensus.cc:2781] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:01.709337 19661 raft_consensus.cc:2454] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7004854aedc24e9e93065d0ee39fb6b3 in term 0.
I20260808 02:21:01.709369 20163 raft_consensus.cc:482] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:01.709395 20163 raft_consensus.cc:3037] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:01.710354 20163 raft_consensus.cc:504] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.710582 20163 leader_election.cc:288] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [CANDIDATE]: Term 1 election: Requested vote from peers dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853), 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409)
I20260808 02:21:01.710768 20072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32942ab3a904f18864c9c294b981194" candidate_uuid: "7004854aedc24e9e93065d0ee39fb6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dada577d1cbd46908d4bce911bafd8fb"
I20260808 02:21:01.710856 20072 raft_consensus.cc:3037] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:01.711054 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32942ab3a904f18864c9c294b981194" candidate_uuid: "7004854aedc24e9e93065d0ee39fb6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b749f373d04a7ca94fbe58035ed119"
I20260808 02:21:01.711134 19661 raft_consensus.cc:3037] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:01.711688 20072 raft_consensus.cc:2454] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7004854aedc24e9e93065d0ee39fb6b3 in term 1.
I20260808 02:21:01.711918 19770 leader_election.cc:302] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb; no voters:
I20260808 02:21:01.711956 19661 raft_consensus.cc:2454] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7004854aedc24e9e93065d0ee39fb6b3 in term 1.
I20260808 02:21:01.712105 20163 raft_consensus.cc:2781] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:01.712311 20163 raft_consensus.cc:686] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 7004854aedc24e9e93065d0ee39fb6b3, State: Running, Role: LEADER
I20260808 02:21:01.712428 20163 consensus_queue.cc:237] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:01.713486 19462 catalog_manager.cc:5495] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130). New cstate: current_term: 1 leader_uuid: "7004854aedc24e9e93065d0ee39fb6b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } health_report { overall_health: UNKNOWN } } }
I20260808 02:21:01.774324 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.771046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.28.177.130:41983)
0808 02:21:01.771181 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:01.771183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:01.771194 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:21:01.771255 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:01.771256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:01.771276 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:01.771370 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:01.771576 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.772132 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.773402 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:01.773622 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:01.773722 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:01.773911 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:01.774030 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:01.774139 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:01.774202 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 02:21:01.774228 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 02:21:01.776547 19661 raft_consensus.cc:1261] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Refusing update from remote peer 7004854aedc24e9e93065d0ee39fb6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:01.776547 20072 raft_consensus.cc:1261] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Refusing update from remote peer 7004854aedc24e9e93065d0ee39fb6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:01.777043 20170 consensus_queue.cc:1035] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:01.777282 20163 consensus_queue.cc:1035] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:01.779527 20182 mvcc.cc:205] Tried to move back new op lower bound from 7316093590632861696 to 7316093590374764544. Current Snapshot: MvccSnapshot[applied={T|T < 7316093590632861696}]
I20260808 02:21:01.780537 20185 mvcc.cc:205] Tried to move back new op lower bound from 7316093590632861696 to 7316093590374764544. Current Snapshot: MvccSnapshot[applied={T|T < 7316093590632861696 or (T in {7316093590632861696})}]
I20260808 02:21:01.782848 19460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:01.782953 19460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:01.783826 19460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Sending DeleteTablet for 3 replicas of tablet d32942ab3a904f18864c9c294b981194
I20260808 02:21:01.784212 19839 tablet_service.cc:1514] Processing DeleteTablet for tablet d32942ab3a904f18864c9c294b981194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:01 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:40394
I20260808 02:21:01.784273 20043 tablet_service.cc:1514] Processing DeleteTablet for tablet d32942ab3a904f18864c9c294b981194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:01 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:42152
I20260808 02:21:01.784523 20193 tablet_replica.cc:330] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: stopping tablet replica
I20260808 02:21:01.784540 20194 tablet_replica.cc:330] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: stopping tablet replica
I20260808 02:21:01.784575 19637 tablet_service.cc:1514] Processing DeleteTablet for tablet d32942ab3a904f18864c9c294b981194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:01 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:48928
I20260808 02:21:01.784648 20194 raft_consensus.cc:2229] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:01.784745 20193 raft_consensus.cc:2229] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:21:01.784765 20194 raft_consensus.cc:2258] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:01.784940 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 19515
I20260808 02:21:01.784948 20193 raft_consensus.cc:2258] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:01.785403 20194 ts_tablet_manager.cc:1905] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:01.789382 20193 ts_tablet_manager.cc:1905] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:01.790292 20194 ts_tablet_manager.cc:1918] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:01.790390 20194 log.cc:1196] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/d32942ab3a904f18864c9c294b981194
I20260808 02:21:01.790700 20194 ts_tablet_manager.cc:1939] T d32942ab3a904f18864c9c294b981194 P dada577d1cbd46908d4bce911bafd8fb: Deleting consensus metadata
I20260808 02:21:01.791200 19438 catalog_manager.cc:4841] TS dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853): tablet d32942ab3a904f18864c9c294b981194 (table test-table [id=1ec815e4ad8b4073b523bad2b6231a3a]) successfully deleted
I20260808 02:21:01.791636 20193 ts_tablet_manager.cc:1918] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:01.791741 20193 log.cc:1196] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/d32942ab3a904f18864c9c294b981194
I20260808 02:21:01.791989 20193 ts_tablet_manager.cc:1939] T d32942ab3a904f18864c9c294b981194 P 7004854aedc24e9e93065d0ee39fb6b3: Deleting consensus metadata
I20260808 02:21:01.792714 19443 catalog_manager.cc:4841] TS 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983): tablet d32942ab3a904f18864c9c294b981194 (table test-table [id=1ec815e4ad8b4073b523bad2b6231a3a]) successfully deleted
W20260808 02:21:01.802366 19443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:33409 (error 108)
I20260808 02:21:01.802680 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 19735
I20260808 02:21:01.803097 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:01.802930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:33409 (local address 127.0.0.1:48932)
0808 02:21:01.803013 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:01.803051 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 02:21:01.803220 19443 catalog_manager.cc:4584] TS 28b749f373d04a7ca94fbe58035ed119 (127.28.177.129:33409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d32942ab3a904f18864c9c294b981194: Network error: Client connection negotiation failed: client connection to 127.28.177.129:33409: connect: Connection refused (error 111)
I20260808 02:21:01.813390 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 19947
I20260808 02:21:01.829094 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 19414
I20260808 02:21:21.840154 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41243
--webserver_interface=127.28.177.190
--webserver_port=33287
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:21.937407 20198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:21.937649 20198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:21.937674 20198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:21.939033 20198 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:21:21.939083 20198 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:21:21.939097 20198 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:21:21.939110 20198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:21.939126 20198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:21.939186 20198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:21.939208 20198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:21.939232 20198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:21.941001 20198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:21.941344 20198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:21.941636 20198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:21.947530 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:21.947625 20207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:21.947659 20198 server_base.cc:1034] running on GCE node
W20260808 02:21:21.947887 20210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:21.948249 20198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:21.948598 20198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:21.949801 20198 hybrid_clock.cc:648] HybridClock initialized: now 1786155681949766 us; error 53 us; skew 500 ppm
Aug 08 02:21:21 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155681, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:21.959060 20198 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:21:21.959640 20198 webserver.cc:460] Webserver started at http://127.28.177.190:33287/ using document root <none> and password file <none>
I20260808 02:21:21.959884 20198 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:21.959971 20198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:21.961534 20198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:21.962504 20217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:21.962769 20198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:21.962894 20198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "d6c399d2fc51444bb3a30a5dddca2799"
format_stamp: "Formatted at 2026-08-08 02:20:39 on dist-test-slave-3h5h"
server_key: "065efddd121fb6443974527231466a4f"
server_key_iv: "cb19085afcf224426143b56ba39c684a"
server_key_version: "encryptionkey@0"
I20260808 02:21:21.963335 20198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:21.980373 20198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:21.983382 20198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:21.983592 20198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:21.988370 20198 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:41243
I20260808 02:21:21.988410 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:41243 every 8 connection(s)
I20260808 02:21:21.988827 20198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:21.990478 20300 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:21:21.991219 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap starting.
I20260808 02:21:21.998560 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 20198
I20260808 02:21:21.998967 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:33409
--local_ip_for_outbound_sockets=127.28.177.129
--tserver_master_addrs=127.28.177.190:41243
--webserver_port=36679
--webserver_interface=127.28.177.129
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:21:22.001744 20300 log.cc:824] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:22.004684 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:21:22.005115 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Bootstrap complete.
I20260808 02:21:22.007579 20300 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:22.007853 20300 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c399d2fc51444bb3a30a5dddca2799, State: Initialized, Role: FOLLOWER
I20260808 02:21:22.008092 20300 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:22.008198 20300 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:22.008257 20300 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:22.008307 20300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:21:22.009495 20300 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:22.009634 20300 leader_election.cc:302] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c399d2fc51444bb3a30a5dddca2799; no voters:
I20260808 02:21:22.009847 20300 leader_election.cc:288] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:21:22.009891 20306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:21:22.010044 20306 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [term 3 LEADER]: Becoming Leader. State: Replica: d6c399d2fc51444bb3a30a5dddca2799, State: Running, Role: LEADER
I20260808 02:21:22.010212 20300 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:22.010217 20306 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } }
I20260808 02:21:22.010798 20311 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c399d2fc51444bb3a30a5dddca2799. Latest consensus state: current_term: 3 leader_uuid: "d6c399d2fc51444bb3a30a5dddca2799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } } }
I20260808 02:21:22.010900 20311 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:22.011229 20307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d6c399d2fc51444bb3a30a5dddca2799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c399d2fc51444bb3a30a5dddca2799" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41243 } } }
I20260808 02:21:22.011379 20307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:22.011269 20318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:22.012442 20318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:22.012741 20318 catalog_manager.cc:1259] Loaded cluster ID: 2731e109ee424054abd66c61fdff3725
I20260808 02:21:22.012784 20318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:22.014005 20318 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:22.014206 20318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Loaded TSK: 0
I20260808 02:21:22.014259 20318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6c399d2fc51444bb3a30a5dddca2799: Loaded TSK: 1
F20260808 02:21:22.014675 20318 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe2fc19b9ad google::LogMessage::Fail() at ??:0
@ 0x7fe2fc19e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe2fc19b4de google::LogMessage::Flush() at ??:0
@ 0x7fe2fc19ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe2fe908104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe2fe905d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe3040f91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe3040f948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe3040f9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe3040bfc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe3040c9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe2fe0690d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe2fe05e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe2feec26db start_thread at ??:0
@ 0x7fe2fac9961f clone at ??:0
W20260808 02:21:22.118294 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:22.118525 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:22.118553 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:22.119882 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:22.119940 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:22.119971 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:22.121752 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:33409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=36679
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:22.122025 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:22.122308 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:22.129186 20372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:22.129258 20376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:22.129212 20371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:22.129621 20303 server_base.cc:1034] running on GCE node
I20260808 02:21:22.129858 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:22.130146 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:22.131341 20303 hybrid_clock.cc:648] HybridClock initialized: now 1786155682131325 us; error 33 us; skew 500 ppm
Aug 08 02:21:22 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155682, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:22.134445 20303 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:21:22.135072 20303 webserver.cc:460] Webserver started at http://127.28.177.129:36679/ using document root <none> and password file <none>
I20260808 02:21:22.135327 20303 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:22.135406 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:22.137080 20303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:22.137902 20385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.138226 20303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:22.138372 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "28b749f373d04a7ca94fbe58035ed119"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "61f971c4647110d527e705f31f3f1c50"
server_key_iv: "752c4e05ba34d7fc5fb1833aee7455eb"
server_key_version: "encryptionkey@0"
I20260808 02:21:22.138772 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:22.156410 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:22.158905 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:22.159178 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:22.159472 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:22.159970 20394 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:21:22.162205 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:21:22.162308 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 02:21:22.162360 20303 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:21:22.163236 20303 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:21:22.163314 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:22.163370 20394 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap starting.
I20260808 02:21:22.171983 20303 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:33409
I20260808 02:21:22.172026 20542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:33409 every 8 connection(s)
I20260808 02:21:22.172477 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:22.175416 20394 log.cc:824] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:22.176987 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 20303
I20260808 02:21:22.177357 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:41983
--local_ip_for_outbound_sockets=127.28.177.130
--tserver_master_addrs=127.28.177.190:41243
--webserver_port=33987
--webserver_interface=127.28.177.130
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:21:22.179409 20394 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:21:22.179749 20394 tablet_bootstrap.cc:492] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Bootstrap complete.
I20260808 02:21:22.181119 20394 ts_tablet_manager.cc:1397] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20260808 02:21:22.182806 20394 raft_consensus.cc:348] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:22.183243 20394 raft_consensus.cc:729] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b749f373d04a7ca94fbe58035ed119, State: Initialized, Role: FOLLOWER
I20260808 02:21:22.183423 20394 consensus_queue.cc:260] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:22.183799 20394 ts_tablet_manager.cc:1428] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260808 02:21:22.227789 20549 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:21:22.173257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:42081)
0808 02:21:22.173465 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.173513 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:22.174020 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:22.227538 (+ 53518us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
W20260808 02:21:22.228143 20543 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:41243 (0 consecutive failures): Network error: Failed to ping master at 127.28.177.190:41243: Client connection negotiation failed: client connection to 127.28.177.190:41243: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:21:22.228543 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.228411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:35367)
0808 02:21:22.228474 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.228521 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:21:22.228960 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.228876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:51917)
0808 02:21:22.228907 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.228932 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260808 02:21:22.229112 20543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:21:22.269371 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:22.269586 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:22.269613 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:22.270736 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:22.270776 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:22.270802 20551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:22.272289 20551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:41983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=33987
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:22.272482 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:22.272688 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:22.278260 20562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:22.278239 20567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:22.278440 20551 server_base.cc:1034] running on GCE node
W20260808 02:21:22.278244 20561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:22.278718 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:22.278970 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:22.280167 20551 hybrid_clock.cc:648] HybridClock initialized: now 1786155682280153 us; error 40 us; skew 500 ppm
Aug 08 02:21:22 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155682, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:22.282980 20551 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:21:22.283413 20551 webserver.cc:460] Webserver started at http://127.28.177.130:33987/ using document root <none> and password file <none>
I20260808 02:21:22.283680 20551 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:22.283732 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:22.285221 20551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:21:22.285893 20576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.286082 20551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:22.286198 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "7004854aedc24e9e93065d0ee39fb6b3"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "ec715513df9c2f82869b44fce5fb8a7a"
server_key_iv: "74c97776436ec2bfef108b6ff17fce34"
server_key_version: "encryptionkey@0"
I20260808 02:21:22.286559 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:22.303983 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:22.306547 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:22.306710 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:22.307000 20551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:22.307348 20551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:22.307404 20551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.307442 20551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:22.307469 20551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.313977 20551 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:41983
I20260808 02:21:22.314034 20755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:41983 every 8 connection(s)
I20260808 02:21:22.314407 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:21:22.315676 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.315230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:43833)
0808 02:21:22.315411 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.315491 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
W20260808 02:21:22.316152 20756 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:41243 (0 consecutive failures): Network error: Failed to ping master at 127.28.177.190:41243: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
I20260808 02:21:22.316161 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 20551
I20260808 02:21:22.316460 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.316354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:33991)
0808 02:21:22.316411 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.316441 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 02:21:22.316509 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:32853
--local_ip_for_outbound_sockets=127.28.177.131
--tserver_master_addrs=127.28.177.190:41243
--webserver_port=41649
--webserver_interface=127.28.177.131
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:21:22.316891 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.316774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:55353)
0808 02:21:22.316823 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.316865 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 02:21:22.317023 20756 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:21:22.424023 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:22.424242 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:22.424268 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:22.425434 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:22.425477 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:22.425503 20760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:22.427045 20760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:32853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=41649
--tserver_master_addrs=127.28.177.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:22.427263 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:22.427448 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:22.433101 20768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:22.433187 20767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:22.433233 20760 server_base.cc:1034] running on GCE node
W20260808 02:21:22.433112 20770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:22.433787 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:22.434114 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:22.435293 20760 hybrid_clock.cc:648] HybridClock initialized: now 1786155682435267 us; error 42 us; skew 500 ppm
Aug 08 02:21:22 dist-test-slave-3h5h krb5kdc[18637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155682, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:22.438567 20760 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:21:22.439076 20760 webserver.cc:460] Webserver started at http://127.28.177.131:41649/ using document root <none> and password file <none>
I20260808 02:21:22.439364 20760 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:22.439420 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:22.441035 20760 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:22.441779 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.442039 20760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:22.442194 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "dada577d1cbd46908d4bce911bafd8fb"
format_stamp: "Formatted at 2026-08-08 02:20:40 on dist-test-slave-3h5h"
server_key: "f3b23238a8fd45496073f9256d4a4b45"
server_key_iv: "3d537b1968e4a4bc406e6de465d81b4e"
server_key_version: "encryptionkey@0"
I20260808 02:21:22.442598 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:22.463984 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:22.466244 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:22.466375 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:22.466603 20760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:22.467012 20760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:22.467056 20760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.467109 20760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:22.467141 20760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:22.473909 20760 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:32853
I20260808 02:21:22.473994 20961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:32853 every 8 connection(s)
I20260808 02:21:22.474396 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:22.475595 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.475186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:45049)
0808 02:21:22.475349 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.475423 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
W20260808 02:21:22.476024 20962 heartbeater.cc:644] Failed to heartbeat to 127.28.177.190:41243 (0 consecutive failures): Network error: Failed to ping master at 127.28.177.190:41243: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
I20260808 02:21:22.476346 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.476249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:54671)
0808 02:21:22.476299 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.476328 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:21:22.476773 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:22.476718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:35161)
0808 02:21:22.476741 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:22.476759 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260808 02:21:22.476926 20962 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:21:22.483188 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 20760
I20260808 02:21:23.229938 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.229604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:34243)
0808 02:21:23.229816 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:23.229866 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260808 02:21:23.317763 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.317495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:54369)
0808 02:21:23.317640 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:23.317693 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260808 02:21:23.477440 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.477230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:51247)
0808 02:21:23.477359 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:23.477389 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260808 02:21:23.895555 20972 raft_consensus.cc:482] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:23.895790 20972 raft_consensus.cc:504] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:23.896270 20972 leader_election.cc:288] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853), 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
Aug 08 02:21:23 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155682, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.130@KRBTEST.COM
Aug 08 02:21:23 dist-test-slave-3h5h krb5kdc[18637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155682, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.131@KRBTEST.COM
I20260808 02:21:23.907051 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.896465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:41983 (local address 127.28.177.129:54485)
0808 02:21:23.896626 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:23.896639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:23.896705 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:23.897639 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:23.897640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:23.898044 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:23.898261 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:23.898267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.899394 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.899395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:23.900210 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:23.900213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.900807 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.900808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:23.900964 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:23.901656 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:23.901668 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:23.903266 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:23.905484 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:23.905491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:23.905494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:23.905727 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:23.905998 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:23.905999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:23.905999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:23.906117 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:23.906510 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:23.906517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:23.906658 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 02:21:23.906765 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 02:21:23.906936 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":77,"thread_start_us":94,"threads_started":1}
I20260808 02:21:23.907064 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.896610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:54485 (local address 127.28.177.130:41983)
0808 02:21:23.896874 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:23.896885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:23.896913 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:21:23.897360 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:23.897362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:23.897408 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:23.897569 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:23.898384 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.899283 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.900313 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.900711 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.900834 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:23.903387 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:23.903391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:23.903392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:23.903403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:23.905377 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:23.905826 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:23.905830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:23.905830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:23.905925 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:23.906184 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:23.906185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:23.906186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:23.906425 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:23.906517 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:23.906817 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 02:21:23.906941 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":65,"threads_started":1}
I20260808 02:21:23.907420 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.896640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53645 (local address 127.28.177.131:32853)
0808 02:21:23.896822 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:23.896881 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:23.896925 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:21:23.897386 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:23.897388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:23.897428 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:23.897607 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:23.898394 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.899350 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.900317 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.900803 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.901028 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:23.903761 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:23.903765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:23.903766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:23.903776 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:23.905630 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:23.906167 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:23.906171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:23.906171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:23.906275 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:23.906538 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:23.906539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:23.906540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:23.906793 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:23.906888 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:23.907142 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 02:21:23.907296 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260808 02:21:23.907308 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:23.896467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:32853 (local address 127.28.177.129:53645)
0808 02:21:23.896634 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:23.896648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:23.896731 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:23.897622 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:23.897626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:23.898054 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:23.898261 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:23.898267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.899470 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.899471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:23.900211 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:23.900213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:23.900989 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:23.900990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:23.901185 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:23.901832 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:23.901836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:23.903639 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:23.905753 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:23.905755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:23.905755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:23.905950 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:23.906364 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:23.906365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:23.906366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:23.906435 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:23.906886 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:23.906887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:23.906955 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 02:21:23.907053 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:21:23.907172 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":21,"thread_start_us":77,"threads_started":1}
W20260808 02:21:23.908596 20386 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
W20260808 02:21:23.908716 20390 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
I20260808 02:21:23.908799 20390 leader_election.cc:302] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28b749f373d04a7ca94fbe58035ed119; no voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb
I20260808 02:21:23.908962 20972 raft_consensus.cc:2726] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:21:24.230813 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:24.230645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:35973)
0808 02:21:24.230715 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:24.230753 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:21:24.318560 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:24.318280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:59385)
0808 02:21:24.318447 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:24.318502 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260808 02:21:24.478214 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:24.477969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:44277)
0808 02:21:24.478123 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:24.478165 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260808 02:21:25.231765 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:25.231433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:44129)
0808 02:21:25.231616 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:25.231692 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260808 02:21:25.319383 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:25.319160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:56647)
0808 02:21:25.319296 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:25.319335 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260808 02:21:25.479070 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:25.478800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:55261)
0808 02:21:25.478982 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:25.479022 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260808 02:21:25.731607 20992 raft_consensus.cc:482] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:25.731714 20992 raft_consensus.cc:504] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:25.732056 20992 leader_election.cc:288] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853), 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
W20260808 02:21:25.732622 20386 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
W20260808 02:21:25.732708 20390 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
I20260808 02:21:25.732813 20390 leader_election.cc:302] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28b749f373d04a7ca94fbe58035ed119; no voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb
I20260808 02:21:25.733057 20992 raft_consensus.cc:2726] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:21:26.232764 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:26.232403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:50647)
0808 02:21:26.232665 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:26.232716 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260808 02:21:26.320251 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:26.320036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:34813)
0808 02:21:26.320169 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:26.320205 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260808 02:21:26.479908 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:26.479697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:55019)
0808 02:21:26.479825 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:26.479862 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260808 02:21:27.233768 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:27.233454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:40375)
0808 02:21:27.233645 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:27.233715 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260808 02:21:27.321261 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:27.320930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:36423)
0808 02:21:27.321170 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:27.321214 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":185,"threads_started":1}
I20260808 02:21:27.480787 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:27.480525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:45419)
0808 02:21:27.480705 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:27.480742 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260808 02:21:28.234730 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:28.234459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:53731)
0808 02:21:28.234627 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:28.234678 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260808 02:21:28.321969 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:28.321728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:38083)
0808 02:21:28.321871 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:28.321909 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260808 02:21:28.350342 21007 raft_consensus.cc:482] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:28.350462 21007 raft_consensus.cc:504] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:28.350755 21007 leader_election.cc:288] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853), 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
W20260808 02:21:28.351222 20390 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
W20260808 02:21:28.351282 20386 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
I20260808 02:21:28.351311 20390 leader_election.cc:302] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28b749f373d04a7ca94fbe58035ed119; no voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb
I20260808 02:21:28.351462 21007 raft_consensus.cc:2726] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:21:28.481549 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:28.481312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:57943)
0808 02:21:28.481455 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:28.481499 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260808 02:21:29.235567 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:29.235317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:50187)
0808 02:21:29.235468 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:29.235517 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260808 02:21:29.322660 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:29.322404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:40137)
0808 02:21:29.322568 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:29.322611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260808 02:21:29.482460 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:29.482171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:45225)
0808 02:21:29.482365 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:29.482407 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260808 02:21:30.236356 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:30.236107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:45175)
0808 02:21:30.236268 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:30.236308 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260808 02:21:30.323496 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:30.323222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:57155)
0808 02:21:30.323387 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:30.323442 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260808 02:21:30.483397 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:30.483111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:58761)
0808 02:21:30.483298 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:30.483344 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260808 02:21:31.237200 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:31.236895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:59123)
0808 02:21:31.237115 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:31.237152 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260808 02:21:31.324394 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:31.324132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:51359)
0808 02:21:31.324299 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:31.324344 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260808 02:21:31.484264 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:31.484013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:38249)
0808 02:21:31.484170 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:31.484214 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260808 02:21:32.238159 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:32.237842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:45129)
0808 02:21:32.238042 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:32.238102 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260808 02:21:32.325273 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:32.325001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:60875)
0808 02:21:32.325184 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:32.325226 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20260808 02:21:32.485224 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:32.484972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:59023)
0808 02:21:32.485134 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:32.485171 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260808 02:21:33.047508 21034 raft_consensus.cc:482] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:33.047672 21034 raft_consensus.cc:504] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada577d1cbd46908d4bce911bafd8fb" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 32853 } } peers { permanent_uuid: "7004854aedc24e9e93065d0ee39fb6b3" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 41983 } } peers { permanent_uuid: "28b749f373d04a7ca94fbe58035ed119" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 33409 } }
I20260808 02:21:33.047873 21034 leader_election.cc:288] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853), 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983)
W20260808 02:21:33.048357 20386 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dada577d1cbd46908d4bce911bafd8fb (127.28.177.131:32853): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
W20260808 02:21:33.048442 20390 leader_election.cc:341] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7004854aedc24e9e93065d0ee39fb6b3 (127.28.177.130:41983): Not found: Tablet not found: d32942ab3a904f18864c9c294b981194
I20260808 02:21:33.048545 20390 leader_election.cc:302] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28b749f373d04a7ca94fbe58035ed119; no voters: 7004854aedc24e9e93065d0ee39fb6b3, dada577d1cbd46908d4bce911bafd8fb
I20260808 02:21:33.048679 21034 raft_consensus.cc:2726] T d32942ab3a904f18864c9c294b981194 P 28b749f373d04a7ca94fbe58035ed119 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:21:33.239110 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:33.238827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:53895)
0808 02:21:33.239011 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:33.239059 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260808 02:21:33.326371 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:33.326112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:53763)
0808 02:21:33.326276 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:33.326323 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260808 02:21:33.486191 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:33.485882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:49213)
0808 02:21:33.486067 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:33.486120 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260808 02:21:34.240051 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:34.239807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:43193)
0808 02:21:34.239956 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:34.240000 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260808 02:21:34.327178 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:34.326896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:42595)
0808 02:21:34.327078 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:34.327127 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260808 02:21:34.487233 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:34.486824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:55521)
0808 02:21:34.487101 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:34.487161 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":72,"threads_started":1}
I20260808 02:21:35.241082 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:35.240677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:56655)
0808 02:21:35.240826 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:35.240935 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260808 02:21:35.328133 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:35.327806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:39529)
0808 02:21:35.327994 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:35.328084 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260808 02:21:35.488173 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:35.487912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:57463)
0808 02:21:35.488074 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:35.488122 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260808 02:21:36.241928 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:36.241680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:60779)
0808 02:21:36.241835 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:36.241879 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260808 02:21:36.329126 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:36.328763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:51377)
0808 02:21:36.329012 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:36.329066 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":158,"threads_started":1}
I20260808 02:21:36.489156 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:36.488922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:53243)
0808 02:21:36.489060 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:36.489106 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260808 02:21:37.242928 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:37.242617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.129:38217)
0808 02:21:37.242832 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:37.242879 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260808 02:21:37.330022 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:37.329805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.130:60137)
0808 02:21:37.329944 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:37.329975 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260808 02:21:37.488260 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 20303
I20260808 02:21:37.497839 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:37.496315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41243 (local address 127.28.177.131:53509)
0808 02:21:37.497707 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:37.497790 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.190:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":83,"threads_started":1}
I20260808 02:21:37.502722 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 20551
I20260808 02:21:37.517925 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 20760
2026-08-08T02:21:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59203 ms)
[----------] 30 tests from SecurityITest (253418 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:37 dist-test-slave-3h5h krb5kdc[21063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:37 dist-test-slave-3h5h krb5kdc[21063](info): set up 2 sockets
Aug 08 02:21:37 dist-test-slave-3h5h krb5kdc[21063](info): commencing operation
krb5kdc: starting...
W20260808 02:21:38.827320 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:38 dist-test-slave-3h5h krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:21:38.972628 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:44351
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:39.063591 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:39.063866 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:39.063910 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:39.065204 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:39.065279 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:39.065308 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:39.065356 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:39.065392 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:39.066984 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:39.067291 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:39.067551 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:39.073472 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.073380 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:39.073536 21088 server_base.cc:1034] running on GCE node
W20260808 02:21:39.073395 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:39.074196 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:39.074542 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:39.075726 21088 hybrid_clock.cc:648] HybridClock initialized: now 1786155699075708 us; error 56 us; skew 500 ppm
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:39.078706 21088 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:21:39.079216 21088 webserver.cc:460] Webserver started at http://127.28.177.190:44057/ using document root <none> and password file <none>
I20260808 02:21:39.079450 21088 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:39.079519 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:39.079685 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:39.080814 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "f51147015ec84ac5ac436ff38714df9e"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "ddf8b4443697be55fd721d2f5091bd55"
server_key_iv: "b1fd0ae3ce16fd479c7000c0e55a211f"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.081285 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "f51147015ec84ac5ac436ff38714df9e"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "ddf8b4443697be55fd721d2f5091bd55"
server_key_iv: "b1fd0ae3ce16fd479c7000c0e55a211f"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.082739 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:21:39.083506 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.083691 21088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:39.083801 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "f51147015ec84ac5ac436ff38714df9e"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "ddf8b4443697be55fd721d2f5091bd55"
server_key_iv: "b1fd0ae3ce16fd479c7000c0e55a211f"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.083880 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:39.094261 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:39.096755 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:39.096964 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:39.101778 21088 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:44351
I20260808 02:21:39.101876 21179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:44351 every 8 connection(s)
I20260808 02:21:39.102466 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:39.103492 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:39.105957 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: Bootstrap starting.
I20260808 02:21:39.106596 21180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:39.106920 21180 log.cc:824] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:39.107678 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: No bootstrap required, opened a new log
I20260808 02:21:39.109475 21180 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51147015ec84ac5ac436ff38714df9e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44351 } }
I20260808 02:21:39.109601 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 21088
I20260808 02:21:39.109649 21180 raft_consensus.cc:374] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:39.109699 21180 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51147015ec84ac5ac436ff38714df9e, State: Initialized, Role: FOLLOWER
I20260808 02:21:39.109730 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:39.109890 21180 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51147015ec84ac5ac436ff38714df9e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44351 } }
I20260808 02:21:39.109949 29382 external_mini_cluster.cc:1428] Setting key f7d29e6e1cbd947fd75837057abb977f
I20260808 02:21:39.109983 21180 raft_consensus.cc:388] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:39.110028 21180 raft_consensus.cc:482] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:39.110080 21180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:39.111150 21180 raft_consensus.cc:504] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51147015ec84ac5ac436ff38714df9e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44351 } }
I20260808 02:21:39.111277 21180 leader_election.cc:302] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f51147015ec84ac5ac436ff38714df9e; no voters:
I20260808 02:21:39.111459 21180 leader_election.cc:288] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:39.111495 21188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:39.111708 21188 raft_consensus.cc:686] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [term 1 LEADER]: Becoming Leader. State: Replica: f51147015ec84ac5ac436ff38714df9e, State: Running, Role: LEADER
I20260808 02:21:39.111754 21180 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:39.112922 21188 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51147015ec84ac5ac436ff38714df9e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44351 } }
I20260808 02:21:39.113585 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51147015ec84ac5ac436ff38714df9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51147015ec84ac5ac436ff38714df9e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44351 } } }
I20260808 02:21:39.113646 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51147015ec84ac5ac436ff38714df9e. Latest consensus state: current_term: 1 leader_uuid: "f51147015ec84ac5ac436ff38714df9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51147015ec84ac5ac436ff38714df9e" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44351 } } }
I20260808 02:21:39.113893 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:39.113900 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e [sys.catalog]: This master's current role is: LEADER
W20260808 02:21:39.114301 21212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:21:39.114372 21212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:21:39.116988 21213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:39.117213 21213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:39.119014 21213 catalog_manager.cc:1347] Generated new cluster ID: 28460a4ec8fd48088cbb7b8704695065
I20260808 02:21:39.119069 21213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:39.121662 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.110556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.28.177.190:44351)
0808 02:21:39.110797 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.110814 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.110846 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:21:39.110941 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.110946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.110988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.111121 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:39.111911 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.112983 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.114358 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.114851 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.114944 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.117735 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:39.117740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:39.117740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:39.117753 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:39.119858 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.120343 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.120346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.120347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.120480 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.120713 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.120714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.120714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.121026 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:39.121174 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.121303 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.121443 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":133,"threads_started":1}
I20260808 02:21:39.134536 21213 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:39.135174 21213 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:39.141490 21213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: Generated new TSK 0
I20260808 02:21:39.141741 21213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:21:39.185222 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44351
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:39.289189 21222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:39.289405 21222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:39.289430 21222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:39.290529 21222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:39.290565 21222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:39.290591 21222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:39.292033 21222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:39.292306 21222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:39.292585 21222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:39.298305 21228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.298442 21234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.298280 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:39.298656 21222 server_base.cc:1034] running on GCE node
I20260808 02:21:39.298916 21222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:39.299216 21222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:39.300458 21222 hybrid_clock.cc:648] HybridClock initialized: now 1786155699300435 us; error 36 us; skew 500 ppm
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:39.303673 21222 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:21:39.304275 21222 webserver.cc:460] Webserver started at http://127.28.177.129:44679/ using document root <none> and password file <none>
I20260808 02:21:39.304522 21222 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:39.304591 21222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:39.304756 21222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:39.305922 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "5b55e29758274af08f444a1c26100b66"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "1736523e4f5ce81548dcfa52db4e274a"
server_key_iv: "6b59147a454f195637a3b4da92287e97"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.306313 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "5b55e29758274af08f444a1c26100b66"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "1736523e4f5ce81548dcfa52db4e274a"
server_key_iv: "6b59147a454f195637a3b4da92287e97"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.307758 21222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:39.308562 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.308763 21222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:39.308889 21222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "5b55e29758274af08f444a1c26100b66"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "1736523e4f5ce81548dcfa52db4e274a"
server_key_iv: "6b59147a454f195637a3b4da92287e97"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.308979 21222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:39.327489 21222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:39.329823 21222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:39.330008 21222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:39.330281 21222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:39.330641 21222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:39.330699 21222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.330744 21222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:39.330777 21222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.337141 21222 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45331
I20260808 02:21:39.337236 21429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45331 every 8 connection(s)
I20260808 02:21:39.337581 21222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:39.343159 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 21222
I20260808 02:21:39.343263 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:21:39.343492 29382 external_mini_cluster.cc:1428] Setting key 3d1c78146576c23f62f6d078f1640d60
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:39.349013 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.338586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:39775 (local address 127.28.177.190:44351)
0808 02:21:39.338723 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.338726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.338909 (+ 183us) server_negotiation.cc:407] Connection header received
0808 02:21:39.339804 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.339806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.339824 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.339875 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:39.341035 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.341586 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.342641 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.342856 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.342943 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.345589 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:39.345591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:39.345591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:39.345600 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:39.347168 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.347697 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.347698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.347699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.347766 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.348061 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.348062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.348063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.348226 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:39.348299 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.348724 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.348862 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:21:39.349313 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.338489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:44351 (local address 127.28.177.129:39775)
0808 02:21:39.338744 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.338800 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.339655 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.339970 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.339974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.340683 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:39.340915 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.340920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.341704 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.341705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.342531 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.342533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.342968 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.342969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.343120 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.343786 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:39.343795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:39.345487 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:39.347263 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:39.347271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:39.347274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:39.347591 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:39.347845 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:39.347846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:39.347846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:39.347988 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:39.348384 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:39.348393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:39.348559 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.348671 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.348820 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260808 02:21:39.350021 21430 heartbeater.cc:347] Connected to a master server at 127.28.177.190:44351
I20260808 02:21:39.350155 21430 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:39.350356 21430 heartbeater.cc:511] Master 127.28.177.190:44351 requested a full tablet report, sending...
I20260808 02:21:39.350943 21133 ts_manager.cc:194] Registered new tserver with Master: 5b55e29758274af08f444a1c26100b66 (127.28.177.129:45331)
I20260808 02:21:39.351656 21133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:39775
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:21:39.377807 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44351
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:39.466290 21440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:39.466516 21440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:39.466539 21440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:39.467803 21440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:39.467856 21440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:39.467885 21440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:39.469614 21440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:39.469889 21440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:39.470180 21440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:39.476367 21449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.476325 21452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.476352 21447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:39.476940 21440 server_base.cc:1034] running on GCE node
I20260808 02:21:39.477308 21440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:39.477679 21440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:39.478874 21440 hybrid_clock.cc:648] HybridClock initialized: now 1786155699478858 us; error 48 us; skew 500 ppm
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:39.481877 21440 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:21:39.482340 21440 webserver.cc:460] Webserver started at http://127.28.177.130:37721/ using document root <none> and password file <none>
I20260808 02:21:39.482561 21440 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:39.482604 21440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:39.482697 21440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:39.483781 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "fb312ec8db4a1f79c6c20e587d6eb9bb"
server_key_iv: "cddb421daaee57f415d3a7fff94eacfd"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.484107 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "fb312ec8db4a1f79c6c20e587d6eb9bb"
server_key_iv: "cddb421daaee57f415d3a7fff94eacfd"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.485581 21440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:39.486749 21462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.486948 21440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:39.487046 21440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "fb312ec8db4a1f79c6c20e587d6eb9bb"
server_key_iv: "cddb421daaee57f415d3a7fff94eacfd"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.487156 21440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:39.505509 21440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:39.508225 21440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:39.508390 21440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:39.508654 21440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:39.509033 21440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:39.509071 21440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.509095 21440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:39.509107 21440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.515673 21440 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:43471
I20260808 02:21:39.515800 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:43471 every 8 connection(s)
I20260808 02:21:39.516124 21440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:39.524547 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 21440
I20260808 02:21:39.524706 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:21:39.524964 29382 external_mini_cluster.cc:1428] Setting key d11b04e2f1603553ece8247257449391
I20260808 02:21:39.526139 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.516966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:48917 (local address 127.28.177.190:44351)
0808 02:21:39.517042 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.517044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.517234 (+ 190us) server_negotiation.cc:407] Connection header received
0808 02:21:39.517779 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.517780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.517791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.517853 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:39.518837 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.519341 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.520353 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.520576 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.520650 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.522962 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:39.522964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:39.522964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:39.522972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:39.524495 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.525112 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.525114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.525116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.525180 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.525454 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.525455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.525455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.525576 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:39.525672 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.525956 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.526048 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:21:39.526369 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.516941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:44351 (local address 127.28.177.130:48917)
0808 02:21:39.517106 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.517151 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.517675 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.517915 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.517919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.518529 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:39.518743 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.518748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.519430 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.519431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.520248 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.520250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.520673 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.520673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.520775 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.521443 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:39.521452 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:39.522865 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:39.524593 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:39.524600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:39.524602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:39.524982 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:39.525266 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:39.525267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:39.525267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:39.525380 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:39.525664 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:39.525671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:39.525821 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.525922 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.526089 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260808 02:21:39.526844 21635 heartbeater.cc:347] Connected to a master server at 127.28.177.190:44351
I20260808 02:21:39.526960 21635 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:39.527155 21635 heartbeater.cc:511] Master 127.28.177.190:44351 requested a full tablet report, sending...
I20260808 02:21:39.527572 21133 ts_manager.cc:194] Registered new tserver with Master: a5fe583ffb0f401dbb560b0ab8e1f0d5 (127.28.177.130:43471)
I20260808 02:21:39.527956 21133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:48917
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:21:39.556350 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44351
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:39.646082 21641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:39.646306 21641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:39.646330 21641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:39.647506 21641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:39.647548 21641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:39.647575 21641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:39.649106 21641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:39.649300 21641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:39.649489 21641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:39.654989 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.655036 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.655025 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:39.655189 21641 server_base.cc:1034] running on GCE node
I20260808 02:21:39.655453 21641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:39.655745 21641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:39.656904 21641 hybrid_clock.cc:648] HybridClock initialized: now 1786155699656884 us; error 28 us; skew 500 ppm
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:39.659551 21641 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:21:39.660064 21641 webserver.cc:460] Webserver started at http://127.28.177.131:33817/ using document root <none> and password file <none>
I20260808 02:21:39.660298 21641 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:39.660367 21641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:39.660527 21641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:39.661672 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "f3920fa00a6445d7aae31d912e316dd1"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "7cf3dfe1c69b8fe9c22ab162ea8199c9"
server_key_iv: "4dde0ae8f39851a9b8b8e20c3af251c5"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.662035 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "f3920fa00a6445d7aae31d912e316dd1"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "7cf3dfe1c69b8fe9c22ab162ea8199c9"
server_key_iv: "4dde0ae8f39851a9b8b8e20c3af251c5"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.663452 21641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:39.664229 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.664417 21641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 02:21:39.664527 21641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "f3920fa00a6445d7aae31d912e316dd1"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "7cf3dfe1c69b8fe9c22ab162ea8199c9"
server_key_iv: "4dde0ae8f39851a9b8b8e20c3af251c5"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.664604 21641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:39.695907 21641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:39.698468 21641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:39.698661 21641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:39.698935 21641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:39.699281 21641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:39.699338 21641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.699379 21641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:39.699409 21641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:39.705888 21641 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:41099
I20260808 02:21:39.705948 21850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:41099 every 8 connection(s)
I20260808 02:21:39.706336 21641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
Aug 08 02:21:39 dist-test-slave-3h5h krb5kdc[21063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:39.713702 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 21641
I20260808 02:21:39.713804 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:21:39.714056 29382 external_mini_cluster.cc:1428] Setting key 56d9f5cbecb1a5c3e8009b48c0abb3e3
I20260808 02:21:39.716675 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.707147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:50897 (local address 127.28.177.190:44351)
0808 02:21:39.707235 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.707237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.707343 (+ 106us) server_negotiation.cc:407] Connection header received
0808 02:21:39.707901 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.707902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.707912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.707966 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:39.709021 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.709534 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.710605 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.710819 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.710900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.713423 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:39.713425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:39.713426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:39.713434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:39.715336 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.715736 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.715737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.715738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.715794 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.716019 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.716021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.716021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.716148 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:39.716217 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.716538 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.716607 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:21:39.716897 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.707067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:44351 (local address 127.28.177.131:50897)
0808 02:21:39.707230 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.707270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.707805 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.708022 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.708026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.708648 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:39.708915 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.708925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.709646 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.709647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.710497 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.710499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.710910 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.710912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.711073 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.711626 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:39.711635 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:39.713307 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:39.715416 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:39.715422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:39.715425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:39.715663 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:39.715842 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:39.715843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:39.715843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:39.715932 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:39.716265 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:39.716272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:39.716414 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.716504 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.716630 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260808 02:21:39.717491 21853 heartbeater.cc:347] Connected to a master server at 127.28.177.190:44351
I20260808 02:21:39.717600 21853 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:39.717811 21853 heartbeater.cc:511] Master 127.28.177.190:44351 requested a full tablet report, sending...
I20260808 02:21:39.718209 21133 ts_manager.cc:194] Registered new tserver with Master: f3920fa00a6445d7aae31d912e316dd1 (127.28.177.131:41099)
I20260808 02:21:39.718547 21133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:50897
I20260808 02:21:39.726339 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:39.734607 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.727937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.28.177.190:44351)
0808 02:21:39.727996 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.727998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.728104 (+ 106us) server_negotiation.cc:407] Connection header received
0808 02:21:39.728136 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.728137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.728147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.728210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:39.728772 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.729465 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.730445 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.730648 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.730715 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.732035 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:39.732036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:39.732036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:39.732044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:39.733554 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.733900 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.733901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.733901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.733958 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:39.734139 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:39.734139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:39.734140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:39.734302 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:39.734380 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.734473 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.734535 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:21:39.736644 21133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:39.738360 21133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:39.745582 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.741174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.28.177.131:41099)
0808 02:21:39.741604 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.741608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.741617 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:21:39.741658 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.741660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.741673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.741740 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:39.741979 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.743306 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.744541 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.745129 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.745209 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.745300 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.745441 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.745488 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":116,"threads_started":1}
I20260808 02:21:39.745599 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.740973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:41099 (local address 127.0.0.1:38176)
0808 02:21:39.741331 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.741382 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.741462 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.741779 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.741783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.741785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:39.741868 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.741871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.743427 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.743428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.744440 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.744442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.745204 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.745205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.745302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.745319 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.745425 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.745503 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":54,"threads_started":1}
I20260808 02:21:39.745602 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.741064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45331 (local address 127.0.0.1:44690)
0808 02:21:39.741400 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.741412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.741471 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.741844 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.741845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.741852 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:39.741922 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.741924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.742989 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.742990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.744208 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.744211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.745338 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.745339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.745440 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.745443 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.745495 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.745540 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":97,"threads_started":1}
I20260808 02:21:39.746044 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.741352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.28.177.129:45331)
0808 02:21:39.741648 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.741652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.741662 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:39.741698 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.741700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.741739 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.741831 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:39.742016 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.742887 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.744311 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.745165 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.745237 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.745301 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.745810 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.745862 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260808 02:21:39.746443 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.741168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:43471 (local address 127.0.0.1:52994)
0808 02:21:39.741965 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.741977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.742036 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.742263 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.742264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.742266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:39.742322 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.742323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.743715 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.743716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.745007 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.745019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.746195 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.746196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.746264 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.746269 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.746320 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.746362 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":88,"threads_started":1}
I20260808 02:21:39.746397 21751 tablet_service.cc:1467] Processing CreateTablet for tablet 51e49bcef8744b2c8f5c62faa786051e (DEFAULT_TABLE table=test-table [id=81300f0e7dad4877aea8424e3006c5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:39.746810 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e49bcef8744b2c8f5c62faa786051e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:39.746898 21323 tablet_service.cc:1467] Processing CreateTablet for tablet 51e49bcef8744b2c8f5c62faa786051e (DEFAULT_TABLE table=test-table [id=81300f0e7dad4877aea8424e3006c5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:39.747236 21323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e49bcef8744b2c8f5c62faa786051e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:39.750609 21877 tablet_bootstrap.cc:492] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: Bootstrap starting.
I20260808 02:21:39.751006 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.741301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.28.177.130:43471)
0808 02:21:39.741512 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.741517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.742111 (+ 594us) server_negotiation.cc:407] Connection header received
0808 02:21:39.742161 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.742164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.742180 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.742268 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:39.742392 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.743553 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.745119 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.746024 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.746096 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.746156 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.750750 (+ 4594us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.750815 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260808 02:21:39.751434 21877 tablet_bootstrap.cc:654] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:39.751664 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 51e49bcef8744b2c8f5c62faa786051e (DEFAULT_TABLE table=test-table [id=81300f0e7dad4877aea8424e3006c5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:39.751976 21537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e49bcef8744b2c8f5c62faa786051e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:39.752048 21877 log.cc:824] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:39.752519 21879 tablet_bootstrap.cc:492] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Bootstrap starting.
I20260808 02:21:39.752991 21877 tablet_bootstrap.cc:492] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: No bootstrap required, opened a new log
I20260808 02:21:39.753069 21877 ts_tablet_manager.cc:1397] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:21:39.753302 21879 tablet_bootstrap.cc:654] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:39.753914 21879 log.cc:824] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:39.754722 21879 tablet_bootstrap.cc:492] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: No bootstrap required, opened a new log
I20260808 02:21:39.754801 21879 ts_tablet_manager.cc:1397] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:39.755066 21887 tablet_bootstrap.cc:492] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Bootstrap starting.
I20260808 02:21:39.755483 21877 raft_consensus.cc:348] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.755697 21877 raft_consensus.cc:374] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:39.755681 21887 tablet_bootstrap.cc:654] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:39.755740 21877 raft_consensus.cc:729] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b55e29758274af08f444a1c26100b66, State: Initialized, Role: FOLLOWER
I20260808 02:21:39.755918 21877 consensus_queue.cc:260] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.756232 21877 ts_tablet_manager.cc:1428] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:39.756366 21887 log.cc:824] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:39.756407 21430 heartbeater.cc:503] Master 127.28.177.190:44351 was elected leader, sending a full tablet report...
I20260808 02:21:39.756947 21879 raft_consensus.cc:348] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.757126 21879 raft_consensus.cc:374] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:39.757182 21879 raft_consensus.cc:729] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3920fa00a6445d7aae31d912e316dd1, State: Initialized, Role: FOLLOWER
I20260808 02:21:39.757223 21887 tablet_bootstrap.cc:492] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: No bootstrap required, opened a new log
I20260808 02:21:39.757304 21887 ts_tablet_manager.cc:1397] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:39.757305 21879 consensus_queue.cc:260] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.757586 21879 ts_tablet_manager.cc:1428] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:39.757615 21853 heartbeater.cc:503] Master 127.28.177.190:44351 was elected leader, sending a full tablet report...
I20260808 02:21:39.759186 21887 raft_consensus.cc:348] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.759315 21887 raft_consensus.cc:374] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:39.759336 21887 raft_consensus.cc:729] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fe583ffb0f401dbb560b0ab8e1f0d5, State: Initialized, Role: FOLLOWER
I20260808 02:21:39.759443 21887 consensus_queue.cc:260] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.759690 21887 ts_tablet_manager.cc:1428] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:21:39.759768 21635 heartbeater.cc:503] Master 127.28.177.190:44351 was elected leader, sending a full tablet report...
W20260808 02:21:39.766734 21636 tablet.cc:2369] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:21:39.785092 21888 raft_consensus.cc:482] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:39.785298 21888 raft_consensus.cc:504] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.785683 21888 leader_election.cc:288] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3920fa00a6445d7aae31d912e316dd1 (127.28.177.131:41099), a5fe583ffb0f401dbb560b0ab8e1f0d5 (127.28.177.130:43471)
I20260808 02:21:39.789302 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.785877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:41099 (local address 127.28.177.129:56739)
0808 02:21:39.785979 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.785992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.786057 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.786231 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.786232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.786234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:39.786328 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.786329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.787242 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.787243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.788112 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.788115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.788972 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.788974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.789075 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.789079 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.789168 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.789214 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":19}
I20260808 02:21:39.789291 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.786010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:53411 (local address 127.28.177.130:43471)
0808 02:21:39.786270 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.786272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.786278 (+ 6us) server_negotiation.cc:407] Connection header received
0808 02:21:39.786326 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.786327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.786337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.786390 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:39.786626 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.787221 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.788309 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.788933 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.788995 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.789003 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.789172 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.789215 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:21:39.789310 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.786042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:56739 (local address 127.28.177.131:41099)
0808 02:21:39.786095 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.786097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.786109 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:21:39.786143 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.786145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.786155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.786219 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:39.786435 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.787154 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.788260 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.788855 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.788927 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.788937 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.789203 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.789245 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:21:39.789321 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.785886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:43471 (local address 127.28.177.129:53411)
0808 02:21:39.786091 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.786102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:39.786171 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:39.786467 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:39.786468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:39.786470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:39.786538 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.786540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.787305 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.787305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.788167 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:39.788170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.789014 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.789014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:39.789074 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.789078 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:39.789121 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 02:21:39.789207 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":15,"thread_start_us":119,"threads_started":1}
I20260808 02:21:39.789726 21781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e49bcef8744b2c8f5c62faa786051e" candidate_uuid: "5b55e29758274af08f444a1c26100b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3920fa00a6445d7aae31d912e316dd1" is_pre_election: true
I20260808 02:21:39.789915 21781 raft_consensus.cc:2454] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b55e29758274af08f444a1c26100b66 in term 0.
I20260808 02:21:39.789898 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e49bcef8744b2c8f5c62faa786051e" candidate_uuid: "5b55e29758274af08f444a1c26100b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" is_pre_election: true
I20260808 02:21:39.790066 21564 raft_consensus.cc:2454] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b55e29758274af08f444a1c26100b66 in term 0.
I20260808 02:21:39.790181 21248 leader_election.cc:302] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b55e29758274af08f444a1c26100b66, f3920fa00a6445d7aae31d912e316dd1; no voters:
I20260808 02:21:39.790316 21888 raft_consensus.cc:2781] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:39.790376 21888 raft_consensus.cc:482] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:39.790410 21888 raft_consensus.cc:3037] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:39.791144 21888 raft_consensus.cc:504] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.791289 21888 leader_election.cc:288] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [CANDIDATE]: Term 1 election: Requested vote from peers f3920fa00a6445d7aae31d912e316dd1 (127.28.177.131:41099), a5fe583ffb0f401dbb560b0ab8e1f0d5 (127.28.177.130:43471)
I20260808 02:21:39.791446 21781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e49bcef8744b2c8f5c62faa786051e" candidate_uuid: "5b55e29758274af08f444a1c26100b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3920fa00a6445d7aae31d912e316dd1"
I20260808 02:21:39.791484 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e49bcef8744b2c8f5c62faa786051e" candidate_uuid: "5b55e29758274af08f444a1c26100b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5"
I20260808 02:21:39.791551 21564 raft_consensus.cc:3037] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:39.791599 21781 raft_consensus.cc:3037] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:39.792313 21564 raft_consensus.cc:2454] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b55e29758274af08f444a1c26100b66 in term 1.
I20260808 02:21:39.792483 21781 raft_consensus.cc:2454] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b55e29758274af08f444a1c26100b66 in term 1.
I20260808 02:21:39.792506 21245 leader_election.cc:302] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b55e29758274af08f444a1c26100b66, a5fe583ffb0f401dbb560b0ab8e1f0d5; no voters:
I20260808 02:21:39.792613 21888 raft_consensus.cc:2781] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:39.792781 21888 raft_consensus.cc:686] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 1 LEADER]: Becoming Leader. State: Replica: 5b55e29758274af08f444a1c26100b66, State: Running, Role: LEADER
I20260808 02:21:39.792920 21888 consensus_queue.cc:237] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } }
I20260808 02:21:39.793785 21133 catalog_manager.cc:5495] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b55e29758274af08f444a1c26100b66 (127.28.177.129). New cstate: current_term: 1 leader_uuid: "5b55e29758274af08f444a1c26100b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b55e29758274af08f444a1c26100b66" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260808 02:21:39.804811 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.802645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.28.177.129:45331)
0808 02:21:39.802701 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:39.802703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:39.802742 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:21:39.802777 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:39.802778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:39.802796 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:39.802854 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:39.802943 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.803413 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.804131 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:39.804325 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:39.804384 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:39.804535 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:39.804637 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:39.804700 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:39.804735 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 02:21:39.804752 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 02:21:39.806748 21564 raft_consensus.cc:1261] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 1 FOLLOWER]: Refusing update from remote peer 5b55e29758274af08f444a1c26100b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:39.806797 21781 raft_consensus.cc:1261] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 1 FOLLOWER]: Refusing update from remote peer 5b55e29758274af08f444a1c26100b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:39.807114 21888 consensus_queue.cc:1035] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fe583ffb0f401dbb560b0ab8e1f0d5" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:39.807330 21895 consensus_queue.cc:1035] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3920fa00a6445d7aae31d912e316dd1" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 41099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:39.809957 21902 mvcc.cc:205] Tried to move back new op lower bound from 7316093746405638144 to 7316093746352316416. Current Snapshot: MvccSnapshot[applied={T|T < 7316093746405638144}]
I20260808 02:21:39.810057 21917 mvcc.cc:205] Tried to move back new op lower bound from 7316093746405638144 to 7316093746352316416. Current Snapshot: MvccSnapshot[applied={T|T < 7316093746405638144}]
I20260808 02:21:39.813354 21132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:39.813505 21132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:39.814522 21132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f51147015ec84ac5ac436ff38714df9e: Sending DeleteTablet for 3 replicas of tablet 51e49bcef8744b2c8f5c62faa786051e
I20260808 02:21:39.814870 21751 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e49bcef8744b2c8f5c62faa786051e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:39 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:38176
I20260808 02:21:39.814867 21323 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e49bcef8744b2c8f5c62faa786051e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:39 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:44690
I20260808 02:21:39.814901 21537 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e49bcef8744b2c8f5c62faa786051e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:39 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:52994
I20260808 02:21:39.815093 21927 tablet_replica.cc:330] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: stopping tablet replica
I20260808 02:21:39.815104 21932 tablet_replica.cc:330] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66: stopping tablet replica
I20260808 02:21:39.815188 21932 raft_consensus.cc:2229] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:21:39.815188 21927 raft_consensus.cc:2229] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:39.815292 21927 raft_consensus.cc:2258] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:39.815301 21932 raft_consensus.cc:2258] T 51e49bcef8744b2c8f5c62faa786051e P 5b55e29758274af08f444a1c26100b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:39.815460 21935 tablet_replica.cc:330] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: stopping tablet replica
I20260808 02:21:39.815572 21935 raft_consensus.cc:2229] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:39.815690 21935 raft_consensus.cc:2258] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:39.815855 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 21222
I20260808 02:21:39.820744 21935 ts_tablet_manager.cc:1905] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:39.822726 21935 ts_tablet_manager.cc:1918] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:39.822810 21935 log.cc:1196] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/51e49bcef8744b2c8f5c62faa786051e
I20260808 02:21:39.823077 21935 ts_tablet_manager.cc:1939] T 51e49bcef8744b2c8f5c62faa786051e P a5fe583ffb0f401dbb560b0ab8e1f0d5: Deleting consensus metadata
I20260808 02:21:39.823575 21110 catalog_manager.cc:4841] TS a5fe583ffb0f401dbb560b0ab8e1f0d5 (127.28.177.130:43471): tablet 51e49bcef8744b2c8f5c62faa786051e (table test-table [id=81300f0e7dad4877aea8424e3006c5a3]) successfully deleted
I20260808 02:21:39.825874 21927 ts_tablet_manager.cc:1905] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:39.827957 21927 ts_tablet_manager.cc:1918] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:39.828053 21927 log.cc:1196] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/51e49bcef8744b2c8f5c62faa786051e
I20260808 02:21:39.828361 21927 ts_tablet_manager.cc:1939] T 51e49bcef8744b2c8f5c62faa786051e P f3920fa00a6445d7aae31d912e316dd1: Deleting consensus metadata
I20260808 02:21:39.828899 21112 catalog_manager.cc:4841] TS f3920fa00a6445d7aae31d912e316dd1 (127.28.177.131:41099): tablet 51e49bcef8744b2c8f5c62faa786051e (table test-table [id=81300f0e7dad4877aea8424e3006c5a3]) successfully deleted
W20260808 02:21:39.830865 21111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:45331 (error 108)
I20260808 02:21:39.831102 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 21440
I20260808 02:21:39.843376 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:39.843201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45331 (local address 127.0.0.1:44710)
0808 02:21:39.843268 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:39.843337 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 02:21:39.843639 21111 catalog_manager.cc:4584] TS 5b55e29758274af08f444a1c26100b66 (127.28.177.129:45331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51e49bcef8744b2c8f5c62faa786051e: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45331: connect: Connection refused (error 111)
I20260808 02:21:39.847626 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 21641
I20260808 02:21:39.863693 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 21088
2026-08-08T02:21:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2342 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T02:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:39Z Disabled control of system clock
I20260808 02:21:39.901360 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:33057
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:39.985170 21943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:39.985410 21943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:39.985431 21943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:39.986634 21943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:39.986681 21943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:39.986694 21943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:39.986707 21943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:39.986721 21943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:39.988271 21943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:33057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:39.988550 21943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:39.988905 21943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:39.994127 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.994156 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:39.994261 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:39.994521 21943 server_base.cc:1034] running on GCE node
I20260808 02:21:39.994789 21943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:39.995090 21943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:39.996304 21943 hybrid_clock.cc:648] HybridClock initialized: now 1786155699996283 us; error 39 us; skew 500 ppm
I20260808 02:21:39.997702 21943 webserver.cc:460] Webserver started at http://127.28.177.190:36309/ using document root <none> and password file <none>
I20260808 02:21:39.997992 21943 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:39.998071 21943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:39.998225 21943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:39.999500 21943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "0317eb438d3247f6ae0e953841bfe794"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "8528e356cadb5bb4274bc515c1a86c42"
server_key_iv: "109a3365143fd3e9dee5c652d71bb670"
server_key_version: "encryptionkey@0"
I20260808 02:21:39.999892 21943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "0317eb438d3247f6ae0e953841bfe794"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "8528e356cadb5bb4274bc515c1a86c42"
server_key_iv: "109a3365143fd3e9dee5c652d71bb670"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.001420 21943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260808 02:21:40.002285 21965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.002477 21943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:40.002602 21943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "0317eb438d3247f6ae0e953841bfe794"
format_stamp: "Formatted at 2026-08-08 02:21:39 on dist-test-slave-3h5h"
server_key: "8528e356cadb5bb4274bc515c1a86c42"
server_key_iv: "109a3365143fd3e9dee5c652d71bb670"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.002678 21943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:40.012514 21943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:40.012955 21943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:40.013129 21943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:40.017576 21943 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:33057
I20260808 02:21:40.017616 22041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:33057 every 8 connection(s)
I20260808 02:21:40.017998 21943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:40.019166 22048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:40.021842 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794: Bootstrap starting.
I20260808 02:21:40.022506 22048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:40.022866 22048 log.cc:824] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:40.023655 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794: No bootstrap required, opened a new log
I20260808 02:21:40.025424 22048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0317eb438d3247f6ae0e953841bfe794" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33057 } }
I20260808 02:21:40.025574 22048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:40.025599 22048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0317eb438d3247f6ae0e953841bfe794, State: Initialized, Role: FOLLOWER
I20260808 02:21:40.025780 22048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0317eb438d3247f6ae0e953841bfe794" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33057 } }
I20260808 02:21:40.025843 22048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:40.025902 22048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:40.025952 22048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:40.026494 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 21943
I20260808 02:21:40.026605 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:40.026650 22048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0317eb438d3247f6ae0e953841bfe794" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33057 } }
I20260808 02:21:40.026769 22048 leader_election.cc:302] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0317eb438d3247f6ae0e953841bfe794; no voters:
I20260808 02:21:40.026829 29382 external_mini_cluster.cc:1428] Setting key af02c97ce0f1719e0d61ef3feb824668
I20260808 02:21:40.026957 22048 leader_election.cc:288] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:40.027019 22053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:40.027191 22053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [term 1 LEADER]: Becoming Leader. State: Replica: 0317eb438d3247f6ae0e953841bfe794, State: Running, Role: LEADER
I20260808 02:21:40.027249 22048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:40.027313 22053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0317eb438d3247f6ae0e953841bfe794" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33057 } }
I20260808 02:21:40.027624 22053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0317eb438d3247f6ae0e953841bfe794. Latest consensus state: current_term: 1 leader_uuid: "0317eb438d3247f6ae0e953841bfe794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0317eb438d3247f6ae0e953841bfe794" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33057 } } }
I20260808 02:21:40.027738 22053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:40.027844 22055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0317eb438d3247f6ae0e953841bfe794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0317eb438d3247f6ae0e953841bfe794" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33057 } } }
I20260808 02:21:40.027966 22055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:40.028541 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:40.029186 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:40.034286 22065 catalog_manager.cc:1347] Generated new cluster ID: 856ba0926e6d4568ab8c5d0d20e9d703
I20260808 02:21:40.034421 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:40.034516 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.027584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.28.177.190:33057)
0808 02:21:40.028069 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.028081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.028120 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:21:40.028251 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.028258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.028332 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.028510 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:40.028955 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.030278 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.031098 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.031549 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.031711 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.032495 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:40.032499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:40.032499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:40.032684 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:40.032724 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:40.032736 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:40.032813 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.033001 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.033089 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":66,"threads_started":1}
I20260808 02:21:40.051873 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:40.052449 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:40.061802 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794: Generated new TSK 0
I20260808 02:21:40.062049 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:40.064755 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:33057
--builtin_ntp_servers=127.28.177.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:40.155949 22086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:40.156209 22086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:40.156235 22086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:40.157696 22086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:40.157752 22086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:40.157781 22086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:40.159389 22086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:33057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:40.159701 22086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:40.160079 22086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:40.165792 22098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:40.165933 22086 server_base.cc:1034] running on GCE node
W20260808 02:21:40.165802 22096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:40.166057 22094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:40.166301 22086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:40.166522 22086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:40.167714 22086 hybrid_clock.cc:648] HybridClock initialized: now 1786155700167701 us; error 41 us; skew 500 ppm
I20260808 02:21:40.169246 22086 webserver.cc:460] Webserver started at http://127.28.177.129:46755/ using document root <none> and password file <none>
I20260808 02:21:40.169529 22086 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:40.169606 22086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:40.169775 22086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:40.170904 22086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "8f6f2d669afa420ea84f06374aa0830f"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "ab7fea560ed869370771550c2dc2d310"
server_key_iv: "f0e100a3c31e48e310279568980410ad"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.171310 22086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6f2d669afa420ea84f06374aa0830f"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "ab7fea560ed869370771550c2dc2d310"
server_key_iv: "f0e100a3c31e48e310279568980410ad"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.172760 22086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:40.173882 22105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.174299 22086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:40.174430 22086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "8f6f2d669afa420ea84f06374aa0830f"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "ab7fea560ed869370771550c2dc2d310"
server_key_iv: "f0e100a3c31e48e310279568980410ad"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.174564 22086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:40.189146 22086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:40.189505 22086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:40.189675 22086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:40.189955 22086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:40.190330 22086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:40.190387 22086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.190424 22086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:40.190471 22086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.196954 22086 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45287
I20260808 02:21:40.197026 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45287 every 8 connection(s)
I20260808 02:21:40.197399 22086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:40.201041 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 22086
I20260808 02:21:40.201162 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:21:40.201366 29382 external_mini_cluster.cc:1428] Setting key 8155c07c24f2431d2d5b7f2607e8f93a
I20260808 02:21:40.202553 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:33057
--builtin_ntp_servers=127.28.177.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:21:40.204144 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.198209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:34051 (local address 127.28.177.190:33057)
0808 02:21:40.198284 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.198286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.198530 (+ 244us) server_negotiation.cc:407] Connection header received
0808 02:21:40.199013 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.199015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.199029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.199089 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:40.199945 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.200541 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.202192 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.202453 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.202519 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.203746 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:40.203747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:40.203747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:40.203753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:40.203769 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:40.203773 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:40.203822 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.203975 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.204054 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:21:40.204391 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.198189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:33057 (local address 127.28.177.129:34051)
0808 02:21:40.198391 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.198427 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.198922 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.199198 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.199201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.199211 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:40.199786 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.199792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.200639 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.200640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.201861 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.201865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.202560 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.202561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.202740 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.203553 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:40.203554 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:40.203596 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:40.203629 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:40.203843 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:40.203851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:40.203854 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.203929 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.204068 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260808 02:21:40.205173 22298 heartbeater.cc:347] Connected to a master server at 127.28.177.190:33057
I20260808 02:21:40.205292 22298 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:40.205507 22298 heartbeater.cc:511] Master 127.28.177.190:33057 requested a full tablet report, sending...
I20260808 02:21:40.206192 21992 ts_manager.cc:194] Registered new tserver with Master: 8f6f2d669afa420ea84f06374aa0830f (127.28.177.129:45287)
I20260808 02:21:40.206830 21992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.129:34051
W20260808 02:21:40.295327 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:40.295663 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:40.295707 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:40.297030 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:40.297120 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:40.297166 22304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:40.298641 22304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:33057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:40.298955 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:40.299316 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:40.305320 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:40.305265 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:40.305323 22304 server_base.cc:1034] running on GCE node
W20260808 02:21:40.305275 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:40.305786 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:40.306131 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:40.307341 22304 hybrid_clock.cc:648] HybridClock initialized: now 1786155700307313 us; error 49 us; skew 500 ppm
I20260808 02:21:40.308655 22304 webserver.cc:460] Webserver started at http://127.28.177.130:35907/ using document root <none> and password file <none>
I20260808 02:21:40.308935 22304 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:40.308997 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:40.309095 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:40.310177 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "ed19e262b7f14fee8e7daf69ebac3f98"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "4854c182d3dbc4da906d8536ffd9aeed"
server_key_iv: "2104336025e564805f552b56377f3c4e"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.310495 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "ed19e262b7f14fee8e7daf69ebac3f98"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "4854c182d3dbc4da906d8536ffd9aeed"
server_key_iv: "2104336025e564805f552b56377f3c4e"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.311867 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:21:40.312676 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.312928 22304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:40.313037 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "ed19e262b7f14fee8e7daf69ebac3f98"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "4854c182d3dbc4da906d8536ffd9aeed"
server_key_iv: "2104336025e564805f552b56377f3c4e"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.313130 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:40.323112 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:40.323413 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:40.323537 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:40.323736 22304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:40.324046 22304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:40.324083 22304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.324107 22304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:40.324129 22304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.330797 22304 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:40879
I20260808 02:21:40.330878 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:40879 every 8 connection(s)
I20260808 02:21:40.331257 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:21:40.336689 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.332190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:45157 (local address 127.28.177.190:33057)
0808 02:21:40.332267 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.332269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.332409 (+ 140us) server_negotiation.cc:407] Connection header received
0808 02:21:40.332964 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.332965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.332976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.333038 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:40.333776 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.334301 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.335283 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.335476 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.335534 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.336398 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:40.336399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:40.336400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:40.336404 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:40.336418 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:40.336422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:40.336467 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.336585 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.336615 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:21:40.336825 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.332069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:33057 (local address 127.28.177.130:45157)
0808 02:21:40.332274 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.332318 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.332807 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.333106 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.333110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.333120 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:40.333669 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.333687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.334392 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.334392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.335194 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.335196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.335560 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.335561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.335676 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.336268 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:40.336269 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:40.336297 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:40.336319 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:40.336479 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:40.336483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:40.336485 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.336538 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.336619 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260808 02:21:40.337432 22509 heartbeater.cc:347] Connected to a master server at 127.28.177.190:33057
I20260808 02:21:40.337546 22509 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:40.337755 22509 heartbeater.cc:511] Master 127.28.177.190:33057 requested a full tablet report, sending...
I20260808 02:21:40.338201 21992 ts_manager.cc:194] Registered new tserver with Master: ed19e262b7f14fee8e7daf69ebac3f98 (127.28.177.130:40879)
I20260808 02:21:40.338665 21992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.130:45157
I20260808 02:21:40.339823 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 22304
I20260808 02:21:40.339923 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:21:40.340102 29382 external_mini_cluster.cc:1428] Setting key 627eeba8f9f1eef0ba47af1cd5f384c7
I20260808 02:21:40.341357 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:33057
--builtin_ntp_servers=127.28.177.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:40.428606 22515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:40.428830 22515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:40.428853 22515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:40.430076 22515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:40.430122 22515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:40.430148 22515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:40.431458 22515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:33057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:40.431654 22515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:40.431911 22515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:40.437304 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:40.437273 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:40.437551 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:40.437836 22515 server_base.cc:1034] running on GCE node
I20260808 02:21:40.438055 22515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:40.438256 22515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:40.439395 22515 hybrid_clock.cc:648] HybridClock initialized: now 1786155700439363 us; error 50 us; skew 500 ppm
I20260808 02:21:40.440697 22515 webserver.cc:460] Webserver started at http://127.28.177.131:37147/ using document root <none> and password file <none>
I20260808 02:21:40.440963 22515 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:40.441012 22515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:40.441111 22515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:40.442143 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "22a47ef233934269a038c94687bde5a9"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "cdff2d8221293cf0fd3bb1aab383ee56"
server_key_iv: "7d946d9c882f58e6c0fa46a3db797116"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.442451 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "22a47ef233934269a038c94687bde5a9"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "cdff2d8221293cf0fd3bb1aab383ee56"
server_key_iv: "7d946d9c882f58e6c0fa46a3db797116"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.443778 22515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:21:40.444543 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.444756 22515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:40.444886 22515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "22a47ef233934269a038c94687bde5a9"
format_stamp: "Formatted at 2026-08-08 02:21:40 on dist-test-slave-3h5h"
server_key: "cdff2d8221293cf0fd3bb1aab383ee56"
server_key_iv: "7d946d9c882f58e6c0fa46a3db797116"
server_key_version: "encryptionkey@0"
I20260808 02:21:40.444975 22515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:40.459192 22515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:40.459487 22515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:40.459594 22515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:40.459802 22515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:40.460105 22515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:40.460139 22515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.460161 22515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:40.460173 22515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:40.466154 22515 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:33287
I20260808 02:21:40.466277 22710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:33287 every 8 connection(s)
I20260808 02:21:40.466588 22515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:40.467926 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 22515
I20260808 02:21:40.468037 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:21:40.468259 29382 external_mini_cluster.cc:1428] Setting key e7d507a80b0316dad7119b8099a9c47c
I20260808 02:21:40.472693 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.467552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:44025 (local address 127.28.177.190:33057)
0808 02:21:40.467645 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.467648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.467903 (+ 255us) server_negotiation.cc:407] Connection header received
0808 02:21:40.468607 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.468608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.468621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.468671 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:40.469616 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.470139 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.471184 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.471378 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.471457 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.472336 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:40.472337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:40.472338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:40.472343 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:40.472364 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:40.472368 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:40.472432 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.472579 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.472612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:21:40.472968 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.467486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:33057 (local address 127.28.177.131:44025)
0808 02:21:40.467754 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.467793 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.468458 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.468748 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.468752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.468761 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:40.469517 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.469524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.470245 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.470247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.471087 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.471090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.471479 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.471480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.471619 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.472185 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:40.472185 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:40.472214 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:40.472240 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:40.472434 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:40.472441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:40.472444 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.472534 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.472649 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260808 02:21:40.473592 22711 heartbeater.cc:347] Connected to a master server at 127.28.177.190:33057
I20260808 02:21:40.473697 22711 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:40.473901 22711 heartbeater.cc:511] Master 127.28.177.190:33057 requested a full tablet report, sending...
I20260808 02:21:40.474375 21992 ts_manager.cc:194] Registered new tserver with Master: 22a47ef233934269a038c94687bde5a9 (127.28.177.131:33287)
I20260808 02:21:40.474749 21992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.131:44025
I20260808 02:21:40.480525 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:40.485715 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.482529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.28.177.190:33057)
0808 02:21:40.482594 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.482596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.482738 (+ 142us) server_negotiation.cc:407] Connection header received
0808 02:21:40.482820 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.482821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.482834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.482887 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:40.483087 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.483587 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.484372 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.484578 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.484668 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.485447 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:40.485449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:40.485449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:40.485454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:40.485470 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:40.485474 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:40.485531 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.485591 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.485617 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:21:40.487352 21992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:40.489143 21992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:40.496043 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.491674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.28.177.130:40879)
0808 02:21:40.492077 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.492083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.492165 (+ 82us) server_negotiation.cc:407] Connection header received
0808 02:21:40.492221 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.492224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.492242 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.492530 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:40.492600 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.493816 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.494983 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.495653 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.495732 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.495825 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.495889 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.495932 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":121,"threads_started":1}
I20260808 02:21:40.495972 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.491560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:40879 (local address 127.0.0.1:45736)
0808 02:21:40.492077 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.492084 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.492146 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.492355 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.492358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.492361 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:40.492443 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.492446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.493925 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.493926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.494899 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.494901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.495736 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.495737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.495794 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.495807 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.495864 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.495906 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260808 02:21:40.496308 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.491523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:33287 (local address 127.0.0.1:48826)
0808 02:21:40.491864 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.491913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.491980 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.492523 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.492524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.492526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:40.492559 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.492560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.493903 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.493904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.495134 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.495137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.496044 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.496045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.496102 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.496105 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.496164 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.496217 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":37,"thread_start_us":97,"threads_started":1}
I20260808 02:21:40.496560 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.491810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.28.177.131:33287)
0808 02:21:40.492351 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.492356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.492365 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:21:40.492401 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.492402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.492416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.492643 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:40.492702 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.493816 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.495230 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.495952 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.496023 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.496108 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.496408 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.496454 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":90,"threads_started":1}
I20260808 02:21:40.497169 22407 tablet_service.cc:1467] Processing CreateTablet for tablet 4464bcf1c36a471cb531933f507e7b4d (DEFAULT_TABLE table=test-table [id=9940dddf41454eb18c61b115bb63b78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:40.497455 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.491523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45287 (local address 127.0.0.1:35824)
0808 02:21:40.491913 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.491926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.492108 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.492425 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.492426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.492428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:40.492624 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.492628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.493743 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.493745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.495134 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.495137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.497226 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.497227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.497277 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.497280 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.497324 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.497355 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":42,"thread_start_us":109,"threads_started":1}
I20260808 02:21:40.497462 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.491839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.28.177.129:45287)
0808 02:21:40.492168 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.492172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.492182 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:40.492224 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.492226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.492242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.492346 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:40.492741 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.493634 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.496150 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.497140 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.497223 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.497309 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.497337 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.497382 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":102,"threads_started":1}
I20260808 02:21:40.497617 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4464bcf1c36a471cb531933f507e7b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:40.498193 22183 tablet_service.cc:1467] Processing CreateTablet for tablet 4464bcf1c36a471cb531933f507e7b4d (DEFAULT_TABLE table=test-table [id=9940dddf41454eb18c61b115bb63b78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:40.498560 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4464bcf1c36a471cb531933f507e7b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:40.500674 22733 tablet_bootstrap.cc:492] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Bootstrap starting.
I20260808 02:21:40.501062 22618 tablet_service.cc:1467] Processing CreateTablet for tablet 4464bcf1c36a471cb531933f507e7b4d (DEFAULT_TABLE table=test-table [id=9940dddf41454eb18c61b115bb63b78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:40.501446 22618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4464bcf1c36a471cb531933f507e7b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:40.501503 22733 tablet_bootstrap.cc:654] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:40.501840 22733 log.cc:824] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:40.502612 22733 tablet_bootstrap.cc:492] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: No bootstrap required, opened a new log
I20260808 02:21:40.502686 22733 ts_tablet_manager.cc:1397] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 02:21:40.504446 22739 tablet_bootstrap.cc:492] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Bootstrap starting.
I20260808 02:21:40.504577 22735 tablet_bootstrap.cc:492] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f: Bootstrap starting.
I20260808 02:21:40.505051 22733 raft_consensus.cc:348] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.505209 22733 raft_consensus.cc:374] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:40.505245 22735 tablet_bootstrap.cc:654] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:40.505255 22733 raft_consensus.cc:729] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed19e262b7f14fee8e7daf69ebac3f98, State: Initialized, Role: FOLLOWER
I20260808 02:21:40.505244 22739 tablet_bootstrap.cc:654] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:40.505393 22733 consensus_queue.cc:260] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.505596 22739 log.cc:824] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:40.505594 22735 log.cc:824] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:40.505676 22733 ts_tablet_manager.cc:1428] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 02:21:40.505834 22509 heartbeater.cc:503] Master 127.28.177.190:33057 was elected leader, sending a full tablet report...
I20260808 02:21:40.506433 22739 tablet_bootstrap.cc:492] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: No bootstrap required, opened a new log
I20260808 02:21:40.506433 22735 tablet_bootstrap.cc:492] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f: No bootstrap required, opened a new log
I20260808 02:21:40.506503 22735 ts_tablet_manager.cc:1397] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:21:40.506503 22739 ts_tablet_manager.cc:1397] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:40.509238 22735 raft_consensus.cc:348] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.509267 22739 raft_consensus.cc:348] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.509428 22735 raft_consensus.cc:374] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:40.509428 22739 raft_consensus.cc:374] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:40.509467 22739 raft_consensus.cc:729] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a47ef233934269a038c94687bde5a9, State: Initialized, Role: FOLLOWER
I20260808 02:21:40.509467 22735 raft_consensus.cc:729] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6f2d669afa420ea84f06374aa0830f, State: Initialized, Role: FOLLOWER
I20260808 02:21:40.509611 22735 consensus_queue.cc:260] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.509742 22739 consensus_queue.cc:260] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.509917 22735 ts_tablet_manager.cc:1428] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:21:40.509989 22739 ts_tablet_manager.cc:1428] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:21:40.510038 22298 heartbeater.cc:503] Master 127.28.177.190:33057 was elected leader, sending a full tablet report...
I20260808 02:21:40.510092 22711 heartbeater.cc:503] Master 127.28.177.190:33057 was elected leader, sending a full tablet report...
I20260808 02:21:40.553479 22743 raft_consensus.cc:482] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:40.553610 22743 raft_consensus.cc:504] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.554071 22743 leader_election.cc:288] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed19e262b7f14fee8e7daf69ebac3f98 (127.28.177.130:40879), 22a47ef233934269a038c94687bde5a9 (127.28.177.131:33287)
I20260808 02:21:40.557404 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.554252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:40879 (local address 127.28.177.129:44373)
0808 02:21:40.554326 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.554340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.554391 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.554574 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.554575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.554578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:40.554665 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.554667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.555486 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.555487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.556404 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.556407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.557133 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.557135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.557208 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.557212 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.557280 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.557329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":29}
I20260808 02:21:40.557430 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.554347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:44373 (local address 127.28.177.130:40879)
0808 02:21:40.554424 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.554427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.554440 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:21:40.554485 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.554487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.554499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.554566 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:40.554759 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.555398 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.556507 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.557052 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.557114 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.557121 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.557309 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.557349 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:21:40.557613 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.554531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:41955 (local address 127.28.177.131:33287)
0808 02:21:40.554579 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.554582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.554785 (+ 203us) server_negotiation.cc:407] Connection header received
0808 02:21:40.554815 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.554816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.554827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.554870 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:40.555049 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.555639 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.556740 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.557229 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.557296 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.557304 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.557487 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.557533 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:21:40.557785 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.554477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:33287 (local address 127.28.177.129:41955)
0808 02:21:40.554672 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.554687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:40.554768 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:40.554893 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:40.554894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:40.554895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:40.554968 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.554970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.555724 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.555725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.556644 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:40.556647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.557319 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.557320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:40.557397 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.557401 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:40.557463 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:21:40.557654 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":140,"threads_started":1}
I20260808 02:21:40.557847 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4464bcf1c36a471cb531933f507e7b4d" candidate_uuid: "8f6f2d669afa420ea84f06374aa0830f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" is_pre_election: true
I20260808 02:21:40.558068 22433 raft_consensus.cc:2454] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6f2d669afa420ea84f06374aa0830f in term 0.
I20260808 02:21:40.558254 22645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4464bcf1c36a471cb531933f507e7b4d" candidate_uuid: "8f6f2d669afa420ea84f06374aa0830f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a47ef233934269a038c94687bde5a9" is_pre_election: true
I20260808 02:21:40.558312 22106 leader_election.cc:302] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f6f2d669afa420ea84f06374aa0830f, ed19e262b7f14fee8e7daf69ebac3f98; no voters:
I20260808 02:21:40.558425 22645 raft_consensus.cc:2454] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6f2d669afa420ea84f06374aa0830f in term 0.
I20260808 02:21:40.558490 22743 raft_consensus.cc:2781] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:40.558563 22743 raft_consensus.cc:482] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:40.558599 22743 raft_consensus.cc:3037] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:40.559396 22743 raft_consensus.cc:504] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.559616 22743 leader_election.cc:288] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [CANDIDATE]: Term 1 election: Requested vote from peers ed19e262b7f14fee8e7daf69ebac3f98 (127.28.177.130:40879), 22a47ef233934269a038c94687bde5a9 (127.28.177.131:33287)
I20260808 02:21:40.559749 22645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4464bcf1c36a471cb531933f507e7b4d" candidate_uuid: "8f6f2d669afa420ea84f06374aa0830f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a47ef233934269a038c94687bde5a9"
I20260808 02:21:40.559739 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4464bcf1c36a471cb531933f507e7b4d" candidate_uuid: "8f6f2d669afa420ea84f06374aa0830f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed19e262b7f14fee8e7daf69ebac3f98"
I20260808 02:21:40.559814 22645 raft_consensus.cc:3037] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:40.559823 22433 raft_consensus.cc:3037] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:40.560531 22645 raft_consensus.cc:2454] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6f2d669afa420ea84f06374aa0830f in term 1.
I20260808 02:21:40.560702 22433 raft_consensus.cc:2454] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6f2d669afa420ea84f06374aa0830f in term 1.
I20260808 02:21:40.560743 22112 leader_election.cc:302] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a47ef233934269a038c94687bde5a9, 8f6f2d669afa420ea84f06374aa0830f; no voters:
I20260808 02:21:40.560840 22743 raft_consensus.cc:2781] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:40.561031 22743 raft_consensus.cc:686] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [term 1 LEADER]: Becoming Leader. State: Replica: 8f6f2d669afa420ea84f06374aa0830f, State: Running, Role: LEADER
I20260808 02:21:40.561175 22743 consensus_queue.cc:237] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } }
I20260808 02:21:40.562198 21989 catalog_manager.cc:5495] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f6f2d669afa420ea84f06374aa0830f (127.28.177.129). New cstate: current_term: 1 leader_uuid: "8f6f2d669afa420ea84f06374aa0830f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f6f2d669afa420ea84f06374aa0830f" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45287 } health_report { overall_health: HEALTHY } } }
I20260808 02:21:40.572819 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.570177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.28.177.129:45287)
0808 02:21:40.570238 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:40.570240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:40.570276 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:21:40.570379 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:40.570384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:40.570407 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:40.570488 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:40.570635 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.571117 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.571947 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:40.572156 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:40.572236 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:40.572408 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:40.572519 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:40.572614 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:40.572644 (+ 30us) server_negotiation.cc:299] Negotiation successful
0808 02:21:40.572675 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:21:40.575065 22645 raft_consensus.cc:1261] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 1 FOLLOWER]: Refusing update from remote peer 8f6f2d669afa420ea84f06374aa0830f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:40.575148 22433 raft_consensus.cc:1261] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 1 FOLLOWER]: Refusing update from remote peer 8f6f2d669afa420ea84f06374aa0830f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:40.575502 22747 consensus_queue.cc:1035] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a47ef233934269a038c94687bde5a9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:40.575632 22743 consensus_queue.cc:1035] T 4464bcf1c36a471cb531933f507e7b4d P 8f6f2d669afa420ea84f06374aa0830f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed19e262b7f14fee8e7daf69ebac3f98" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:40.578756 22752 mvcc.cc:205] Tried to move back new op lower bound from 7316093749552087040 to 7316093749499363328. Current Snapshot: MvccSnapshot[applied={T|T < 7316093749552087040}]
I20260808 02:21:40.579164 22750 mvcc.cc:205] Tried to move back new op lower bound from 7316093749552087040 to 7316093749499363328. Current Snapshot: MvccSnapshot[applied={T|T < 7316093749552087040}]
I20260808 02:21:40.581439 21989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:40.581535 21989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:40.582479 21989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0317eb438d3247f6ae0e953841bfe794: Sending DeleteTablet for 3 replicas of tablet 4464bcf1c36a471cb531933f507e7b4d
I20260808 02:21:40.582850 22618 tablet_service.cc:1514] Processing DeleteTablet for tablet 4464bcf1c36a471cb531933f507e7b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:40 UTC) from {username='slave'} at 127.0.0.1:48826
I20260808 02:21:40.582868 22183 tablet_service.cc:1514] Processing DeleteTablet for tablet 4464bcf1c36a471cb531933f507e7b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:40 UTC) from {username='slave'} at 127.0.0.1:35824
I20260808 02:21:40.583122 22764 tablet_replica.cc:330] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: stopping tablet replica
I20260808 02:21:40.583142 22407 tablet_service.cc:1514] Processing DeleteTablet for tablet 4464bcf1c36a471cb531933f507e7b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:40 UTC) from {username='slave'} at 127.0.0.1:45736
I20260808 02:21:40.583228 22764 raft_consensus.cc:2229] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:40.583344 22764 raft_consensus.cc:2258] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:40.583463 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 22086
I20260808 02:21:40.591141 22764 ts_tablet_manager.cc:1905] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:40.593080 22770 tablet_replica.cc:330] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: stopping tablet replica
I20260808 02:21:40.593266 22770 raft_consensus.cc:2229] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:40.593457 22770 raft_consensus.cc:2258] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:40.593526 22764 ts_tablet_manager.cc:1918] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:40.593623 22764 log.cc:1196] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/4464bcf1c36a471cb531933f507e7b4d
I20260808 02:21:40.593911 22764 ts_tablet_manager.cc:1939] T 4464bcf1c36a471cb531933f507e7b4d P 22a47ef233934269a038c94687bde5a9: Deleting consensus metadata
I20260808 02:21:40.594067 22770 ts_tablet_manager.cc:1905] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:40.594440 21969 catalog_manager.cc:4841] TS 22a47ef233934269a038c94687bde5a9 (127.28.177.131:33287): tablet 4464bcf1c36a471cb531933f507e7b4d (table test-table [id=9940dddf41454eb18c61b115bb63b78e]) successfully deleted
I20260808 02:21:40.595834 22770 ts_tablet_manager.cc:1918] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:40.595932 22770 log.cc:1196] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/4464bcf1c36a471cb531933f507e7b4d
I20260808 02:21:40.596221 22770 ts_tablet_manager.cc:1939] T 4464bcf1c36a471cb531933f507e7b4d P ed19e262b7f14fee8e7daf69ebac3f98: Deleting consensus metadata
I20260808 02:21:40.596767 21966 catalog_manager.cc:4841] TS ed19e262b7f14fee8e7daf69ebac3f98 (127.28.177.130:40879): tablet 4464bcf1c36a471cb531933f507e7b4d (table test-table [id=9940dddf41454eb18c61b115bb63b78e]) successfully deleted
W20260808 02:21:40.597429 21970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:45287 (error 108)
I20260808 02:21:40.597680 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 22304
I20260808 02:21:40.599936 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:40.597841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45287 (local address 127.0.0.1:35832)
0808 02:21:40.599839 (+ 1998us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:40.599904 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1962}
W20260808 02:21:40.600139 21970 catalog_manager.cc:4584] TS 8f6f2d669afa420ea84f06374aa0830f (127.28.177.129:45287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4464bcf1c36a471cb531933f507e7b4d: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45287: connect: Connection refused (error 111)
I20260808 02:21:40.608664 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 22515
I20260808 02:21:40.619546 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 21943
2026-08-08T02:21:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (754 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:40 dist-test-slave-3h5h krb5kdc[22779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:40 dist-test-slave-3h5h krb5kdc[22779](info): set up 2 sockets
Aug 08 02:21:40 dist-test-slave-3h5h krb5kdc[22779](info): commencing operation
krb5kdc: starting...
W20260808 02:21:41.879381 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:41 dist-test-slave-3h5h krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:21:42.025297 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42925
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:42.116748 22797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:42.116993 22797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:42.117017 22797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:42.118199 22797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:42.118245 22797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:42.118260 22797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:42.118273 22797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:42.118285 22797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:42.119812 22797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:42.120070 22797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:42.120333 22797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:42.125883 22805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:42.125936 22807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:42.126230 22797 server_base.cc:1034] running on GCE node
W20260808 02:21:42.125883 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:42.126660 22797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:42.126983 22797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:42.128170 22797 hybrid_clock.cc:648] HybridClock initialized: now 1786155702128187 us; error 80 us; skew 500 ppm
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:42.130853 22797 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:21:42.131409 22797 webserver.cc:460] Webserver started at http://127.28.177.190:45711/ using document root <none> and password file <none>
I20260808 02:21:42.131654 22797 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:42.131723 22797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:42.131897 22797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:42.133090 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "cd52de0631e7489fb50c219dda2f10d5"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "c0ff914cf3b399d44b6e6fbae2ef4022"
server_key_iv: "54e7b6d0aeaea8c3fc5841d9e756afcb"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.133471 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "cd52de0631e7489fb50c219dda2f10d5"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "c0ff914cf3b399d44b6e6fbae2ef4022"
server_key_iv: "54e7b6d0aeaea8c3fc5841d9e756afcb"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.134933 22797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:42.135921 22819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.136255 22797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:42.136374 22797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "cd52de0631e7489fb50c219dda2f10d5"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "c0ff914cf3b399d44b6e6fbae2ef4022"
server_key_iv: "54e7b6d0aeaea8c3fc5841d9e756afcb"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.136456 22797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:42.156736 22797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:42.159327 22797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:42.159548 22797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:42.164486 22797 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42925
I20260808 02:21:42.164518 22911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42925 every 8 connection(s)
I20260808 02:21:42.164991 22797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:42.166071 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:42.168766 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5: Bootstrap starting.
I20260808 02:21:42.169492 22912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:42.169790 22912 log.cc:824] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:42.170648 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5: No bootstrap required, opened a new log
I20260808 02:21:42.172590 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 22797
I20260808 02:21:42.172726 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:42.172577 22912 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd52de0631e7489fb50c219dda2f10d5" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42925 } }
I20260808 02:21:42.172809 22912 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:42.172891 22912 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd52de0631e7489fb50c219dda2f10d5, State: Initialized, Role: FOLLOWER
I20260808 02:21:42.173002 29382 external_mini_cluster.cc:1428] Setting key ead5bb66d999b3fe61444590c8c56a08
I20260808 02:21:42.173095 22912 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd52de0631e7489fb50c219dda2f10d5" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42925 } }
I20260808 02:21:42.173199 22912 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:42.173252 22912 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:42.173313 22912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:42.174659 22912 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd52de0631e7489fb50c219dda2f10d5" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42925 } }
I20260808 02:21:42.174762 22912 leader_election.cc:302] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd52de0631e7489fb50c219dda2f10d5; no voters:
I20260808 02:21:42.174970 22912 leader_election.cc:288] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:42.175034 22924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:42.175197 22924 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [term 1 LEADER]: Becoming Leader. State: Replica: cd52de0631e7489fb50c219dda2f10d5, State: Running, Role: LEADER
I20260808 02:21:42.175302 22912 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:42.175300 22924 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd52de0631e7489fb50c219dda2f10d5" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42925 } }
I20260808 02:21:42.176088 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd52de0631e7489fb50c219dda2f10d5. Latest consensus state: current_term: 1 leader_uuid: "cd52de0631e7489fb50c219dda2f10d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd52de0631e7489fb50c219dda2f10d5" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42925 } } }
I20260808 02:21:42.176201 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:42.176549 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd52de0631e7489fb50c219dda2f10d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd52de0631e7489fb50c219dda2f10d5" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42925 } } }
I20260808 02:21:42.176674 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:42.177074 22939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:42.177582 22939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:42.179291 22939 catalog_manager.cc:1347] Generated new cluster ID: 29b7657905b9451f84c2db7a6d7c09b6
I20260808 02:21:42.179342 22939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:42.187706 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.173562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.28.177.190:42925)
0808 02:21:42.173841 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.173853 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.173895 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:21:42.174058 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.174069 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.174147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.174355 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:42.175136 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.177993 (+ 2857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.179094 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.179712 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.179811 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.182762 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:42.182767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:42.182768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:42.182782 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:42.185774 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.186269 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.186272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.186273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.186406 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.186641 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.186642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.186642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.186994 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:42.187178 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.187301 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.187459 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":129,"threads_started":1}
I20260808 02:21:42.194762 22939 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:42.195338 22939 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:42.213768 22939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5: Generated new TSK 0
I20260808 02:21:42.214021 22939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:21:42.250267 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42925
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:42.337534 22958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:42.337749 22958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:42.337774 22958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:42.338862 22958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:42.338896 22958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:42.338920 22958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:42.340497 22958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:42.340744 22958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:42.340955 22958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:42.346619 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:42.346694 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:42.346623 22974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:42.346952 22958 server_base.cc:1034] running on GCE node
I20260808 02:21:42.347153 22958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:42.347390 22958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:42.348577 22958 hybrid_clock.cc:648] HybridClock initialized: now 1786155702348555 us; error 30 us; skew 500 ppm
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:42.356365 22958 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:21:42.356964 22958 webserver.cc:460] Webserver started at http://127.28.177.129:43283/ using document root <none> and password file <none>
I20260808 02:21:42.357229 22958 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:42.357302 22958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:42.357462 22958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:42.358573 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "1a464db250934e0f9396f5b3abd537f8"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "a7db80c3b2c17c9c38dd0024d66aecc3"
server_key_iv: "5e9f5de17d685c16b179909b3f84698b"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.358958 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "1a464db250934e0f9396f5b3abd537f8"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "a7db80c3b2c17c9c38dd0024d66aecc3"
server_key_iv: "5e9f5de17d685c16b179909b3f84698b"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.360503 22958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:42.361438 22984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.361698 22958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:42.361815 22958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "1a464db250934e0f9396f5b3abd537f8"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "a7db80c3b2c17c9c38dd0024d66aecc3"
server_key_iv: "5e9f5de17d685c16b179909b3f84698b"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.361894 22958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:42.381224 22958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:42.383760 22958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:42.383944 22958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:42.384229 22958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:42.384581 22958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:42.384642 22958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.384688 22958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:42.384711 22958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.391089 22958 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:45513
I20260808 02:21:42.391151 23156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:45513 every 8 connection(s)
I20260808 02:21:42.391484 22958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:42.396782 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 22958
I20260808 02:21:42.396951 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:21:42.397265 29382 external_mini_cluster.cc:1428] Setting key 8df1aae998eb56b612f72a0efc40c6e9
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:42.402243 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.392294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42925 (local address 127.28.177.129:36609)
0808 02:21:42.392485 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.392529 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.393085 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.393305 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.393309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.394009 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:42.394243 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.394249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.394933 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.394935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.395782 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.395785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.396185 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.396186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.396309 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.396902 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:42.396917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:42.398852 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:42.400614 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:42.400623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:42.400625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:42.400992 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:42.401279 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:42.401280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:42.401281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:42.401424 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:42.401731 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:42.401737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:42.401859 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.401941 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.402069 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260808 02:21:42.402256 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.392332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:36609 (local address 127.28.177.190:42925)
0808 02:21:42.392388 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.392390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.392664 (+ 274us) server_negotiation.cc:407] Connection header received
0808 02:21:42.393171 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.393172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.393186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.393243 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:42.394334 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.394839 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.395871 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.396061 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.396119 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.398954 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:42.398956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:42.398956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:42.398964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:42.400524 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.401115 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.401116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.401116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.401199 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.401504 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.401506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.401506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.401665 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:42.401735 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.402006 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.402115 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:21:42.402971 23160 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42925
I20260808 02:21:42.403092 23160 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:42.403311 23160 heartbeater.cc:511] Master 127.28.177.190:42925 requested a full tablet report, sending...
I20260808 02:21:42.403885 22849 ts_manager.cc:194] Registered new tserver with Master: 1a464db250934e0f9396f5b3abd537f8 (127.28.177.129:45513)
I20260808 02:21:42.404500 22849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:36609
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:21:42.428450 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42925
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:42.538079 23169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:42.538312 23169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:42.538337 23169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:42.539724 23169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:42.539773 23169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:42.539805 23169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:42.541530 23169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:42.541793 23169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:42.542052 23169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:42.547598 23179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:42.547842 23169 server_base.cc:1034] running on GCE node
W20260808 02:21:42.547612 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:42.547631 23175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:42.548349 23169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:42.548606 23169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:42.549773 23169 hybrid_clock.cc:648] HybridClock initialized: now 1786155702549741 us; error 56 us; skew 500 ppm
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:42.552524 23169 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:21:42.553061 23169 webserver.cc:460] Webserver started at http://127.28.177.130:39661/ using document root <none> and password file <none>
I20260808 02:21:42.553297 23169 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:42.553369 23169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:42.553519 23169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:42.554661 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "7d3f2bd2978a63eee4142eecd71d42c9"
server_key_iv: "c7bf8d00ee51c5c709d3f8b345d27f40"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.555039 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "7d3f2bd2978a63eee4142eecd71d42c9"
server_key_iv: "c7bf8d00ee51c5c709d3f8b345d27f40"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.556522 23169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:42.557574 23195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.557781 23169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:42.557894 23169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "7d3f2bd2978a63eee4142eecd71d42c9"
server_key_iv: "c7bf8d00ee51c5c709d3f8b345d27f40"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.557976 23169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:42.575350 23169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:42.577898 23169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:42.578068 23169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:42.578318 23169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:42.578657 23169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:42.578713 23169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.578748 23169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:42.578776 23169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.585013 23169 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:36141
I20260808 02:21:42.585086 23370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:36141 every 8 connection(s)
I20260808 02:21:42.585397 23169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:21:42.587778 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 23169
I20260808 02:21:42.588064 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:21:42.588347 29382 external_mini_cluster.cc:1428] Setting key 571501f8bda049c4ce3e04c6fd3768e3
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
I20260808 02:21:42.602891 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.586166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:52303 (local address 127.28.177.190:42925)
0808 02:21:42.586310 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.586313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.586599 (+ 286us) server_negotiation.cc:407] Connection header received
0808 02:21:42.587280 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.587282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.587296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.587361 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:42.588674 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.589574 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.590852 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.591204 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.591306 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.599173 (+ 7867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:42.599176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:42.599176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:42.599186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:42.600774 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.601376 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.601378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.601379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.601441 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.601807 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.601809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.601809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.602005 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:42.602151 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.602712 (+ 561us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.602812 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:21:42.603193 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.586151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42925 (local address 127.28.177.130:52303)
0808 02:21:42.586379 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.586439 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.587151 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.587444 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.587449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.588257 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:42.588516 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.588524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.589705 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.589706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.590725 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.590728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.591349 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.591350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.591534 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.596969 (+ 5435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:42.596986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:42.599015 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:42.600985 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:42.600995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:42.600998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:42.601289 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:42.601534 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:42.601535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:42.601536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:42.601698 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:42.602194 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:42.602202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:42.602481 (+ 279us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.602652 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.602884 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260808 02:21:42.604215 23371 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42925
I20260808 02:21:42.604403 23371 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:42.604704 23371 heartbeater.cc:511] Master 127.28.177.190:42925 requested a full tablet report, sending...
I20260808 02:21:42.605319 22849 ts_manager.cc:194] Registered new tserver with Master: 4590fbe5b2f54b8fa61d82b74bb3c8ca (127.28.177.130:36141)
I20260808 02:21:42.605898 22849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:52303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:21:42.621830 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42925
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:42.708678 23379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:42.709048 23379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:42.709079 23379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:42.710292 23379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:42.710335 23379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:42.710361 23379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:42.711869 23379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:42.712112 23379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:42.712364 23379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:42.717938 23386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:42.717900 23390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:42.717914 23387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:42.718349 23379 server_base.cc:1034] running on GCE node
I20260808 02:21:42.718611 23379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:42.718926 23379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:42.720271 23379 hybrid_clock.cc:648] HybridClock initialized: now 1786155702720239 us; error 39 us; skew 500 ppm
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:42.723182 23379 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:21:42.723591 23379 webserver.cc:460] Webserver started at http://127.28.177.131:46577/ using document root <none> and password file <none>
I20260808 02:21:42.723781 23379 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:42.723821 23379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:42.723913 23379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:42.725018 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "503839bee78841d3999ccea15317bd96"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "2e26a7c9c806a33712d4b5ec0c97b322"
server_key_iv: "af2be3d8dd9b9b1d365c5ebcc42c14b2"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.725347 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "503839bee78841d3999ccea15317bd96"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "2e26a7c9c806a33712d4b5ec0c97b322"
server_key_iv: "af2be3d8dd9b9b1d365c5ebcc42c14b2"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.726665 23379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:21:42.727437 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.727607 23379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.727717 23379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "503839bee78841d3999ccea15317bd96"
format_stamp: "Formatted at 2026-08-08 02:21:42 on dist-test-slave-3h5h"
server_key: "2e26a7c9c806a33712d4b5ec0c97b322"
server_key_iv: "af2be3d8dd9b9b1d365c5ebcc42c14b2"
server_key_version: "encryptionkey@0"
I20260808 02:21:42.727802 23379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:42.734913 23379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:42.736764 23379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:42.737020 23379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:42.737298 23379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:42.737627 23379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:42.737686 23379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.737730 23379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:42.737758 23379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:42.744500 23379 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:36921
I20260808 02:21:42.744567 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:36921 every 8 connection(s)
I20260808 02:21:42.744987 23379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:42.748176 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 23379
I20260808 02:21:42.748328 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:21:42.748540 29382 external_mini_cluster.cc:1428] Setting key 040c8de3e22c891d38fe9fc626bd9908
Aug 08 02:21:42 dist-test-slave-3h5h krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155702, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:42.755201 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.745817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:36299 (local address 127.28.177.190:42925)
0808 02:21:42.745906 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.745908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.746121 (+ 213us) server_negotiation.cc:407] Connection header received
0808 02:21:42.746788 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.746790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.746805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.746889 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:42.748109 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.748782 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.750144 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.750364 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.750432 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.752576 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:42.752577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:42.752577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:42.752585 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:42.753972 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.754351 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.754352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.754353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.754411 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.754628 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.754628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.754629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.754769 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:42.754825 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.755062 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.755130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:21:42.755425 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.745720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42925 (local address 127.28.177.131:36299)
0808 02:21:42.745943 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.746011 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.746642 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.746964 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.746977 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.747709 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:42.747960 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.747978 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.748962 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.748964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.750023 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.750026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.750451 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.750452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.750556 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.751085 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:42.751094 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:42.752477 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:42.754048 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:42.754055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:42.754057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:42.754276 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:42.754475 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:42.754476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:42.754476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:42.754572 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:42.754829 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:42.754834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:42.754949 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.755033 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.755152 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":118,"threads_started":1}
I20260808 02:21:42.755931 23579 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42925
I20260808 02:21:42.756021 23579 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:42.756198 23579 heartbeater.cc:511] Master 127.28.177.190:42925 requested a full tablet report, sending...
I20260808 02:21:42.756564 22849 ts_manager.cc:194] Registered new tserver with Master: 503839bee78841d3999ccea15317bd96 (127.28.177.131:36921)
I20260808 02:21:42.756968 22849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:36299
I20260808 02:21:42.760790 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:42.768671 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.762358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.28.177.190:42925)
0808 02:21:42.762399 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.762401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.762508 (+ 107us) server_negotiation.cc:407] Connection header received
0808 02:21:42.762623 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.762623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.762634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.762677 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:42.763238 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.763741 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.764488 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.764759 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.764817 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.766254 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:42.766255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:42.766255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:42.766262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:42.767588 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.767968 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.767969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.767970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.768024 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:42.768210 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:42.768210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:42.768211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:42.768341 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:42.768420 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.768517 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.768605 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:21:42.770350 22849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:42.772056 22849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:42.779726 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.775500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.28.177.130:36141)
0808 02:21:42.775697 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.775703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.775713 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:42.775756 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.775758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.775774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.775854 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:42.776097 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.777214 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.778784 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.779275 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.779372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.779433 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.779591 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.779630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260808 02:21:42.779747 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.775388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:36141 (local address 127.0.0.1:38470)
0808 02:21:42.775530 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.775541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.775593 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.775941 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.775942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.775945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:42.775998 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.775999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.777316 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.777317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.778676 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.778679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.779370 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.779371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.779450 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.779466 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.779558 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.779623 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":77,"thread_start_us":79,"threads_started":1}
I20260808 02:21:42.779961 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.774918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45513 (local address 127.0.0.1:51404)
0808 02:21:42.775049 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.775092 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.775198 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.775563 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.775567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.775569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:42.775653 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.775655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.777189 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.777190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.778486 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.778490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.779483 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.779484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.779585 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.779589 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.779639 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.779681 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":215,"thread_start_us":78,"threads_started":1}
I20260808 02:21:42.780158 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.775036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.28.177.129:45513)
0808 02:21:42.775334 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.775339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.775351 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:21:42.775394 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.775396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.775413 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.775712 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:42.775806 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.777075 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.778592 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.779315 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.779392 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.779443 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.779793 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.779847 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":148,"threads_started":1}
I20260808 02:21:42.780177 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.775167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:36921 (local address 127.0.0.1:41858)
0808 02:21:42.775541 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.775560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.775606 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.775998 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.775998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.776000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:42.776054 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.776055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.777124 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.777125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.778689 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.778691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.779649 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.779650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.779712 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.779716 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.779763 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.780103 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":80,"thread_start_us":85,"threads_started":1}
I20260808 02:21:42.780613 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.775285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.28.177.131:36921)
0808 02:21:42.775768 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.775771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.775781 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:42.775828 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.775830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.775845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.775917 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:42.776133 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.777007 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.778777 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.779499 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.779566 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.779614 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.779959 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.780535 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":111,"threads_started":1}
I20260808 02:21:42.780800 23267 tablet_service.cc:1467] Processing CreateTablet for tablet 50e1e6925e6d4b4a990b6fa3d6defedf (DEFAULT_TABLE table=test-table [id=ec0b824d90114c16bb006eb32e922ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:42.780974 23055 tablet_service.cc:1467] Processing CreateTablet for tablet 50e1e6925e6d4b4a990b6fa3d6defedf (DEFAULT_TABLE table=test-table [id=ec0b824d90114c16bb006eb32e922ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:42.781277 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e1e6925e6d4b4a990b6fa3d6defedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:42.781412 23055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e1e6925e6d4b4a990b6fa3d6defedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:42.784004 23609 tablet_bootstrap.cc:492] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: Bootstrap starting.
I20260808 02:21:42.784672 23609 tablet_bootstrap.cc:654] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:42.784672 23478 tablet_service.cc:1467] Processing CreateTablet for tablet 50e1e6925e6d4b4a990b6fa3d6defedf (DEFAULT_TABLE table=test-table [id=ec0b824d90114c16bb006eb32e922ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:42.785020 23611 tablet_bootstrap.cc:492] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8: Bootstrap starting.
I20260808 02:21:42.785096 23478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e1e6925e6d4b4a990b6fa3d6defedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:42.785303 23609 log.cc:824] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:42.785686 23611 tablet_bootstrap.cc:654] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:42.786078 23609 tablet_bootstrap.cc:492] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: No bootstrap required, opened a new log
I20260808 02:21:42.786168 23609 ts_tablet_manager.cc:1397] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:42.786238 23611 log.cc:824] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:42.787607 23611 tablet_bootstrap.cc:492] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8: No bootstrap required, opened a new log
I20260808 02:21:42.787681 23611 ts_tablet_manager.cc:1397] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:21:42.788074 23609 raft_consensus.cc:348] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.788247 23609 raft_consensus.cc:374] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:42.788292 23609 raft_consensus.cc:729] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4590fbe5b2f54b8fa61d82b74bb3c8ca, State: Initialized, Role: FOLLOWER
I20260808 02:21:42.788448 23609 consensus_queue.cc:260] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.788787 23609 ts_tablet_manager.cc:1428] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:42.788822 23371 heartbeater.cc:503] Master 127.28.177.190:42925 was elected leader, sending a full tablet report...
I20260808 02:21:42.789039 23615 tablet_bootstrap.cc:492] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Bootstrap starting.
I20260808 02:21:42.789721 23615 tablet_bootstrap.cc:654] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:42.789673 23611 raft_consensus.cc:348] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.789861 23611 raft_consensus.cc:374] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:42.789917 23611 raft_consensus.cc:729] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a464db250934e0f9396f5b3abd537f8, State: Initialized, Role: FOLLOWER
I20260808 02:21:42.790045 23611 consensus_queue.cc:260] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.790355 23611 ts_tablet_manager.cc:1428] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:42.790359 23615 log.cc:824] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:42.790503 23160 heartbeater.cc:503] Master 127.28.177.190:42925 was elected leader, sending a full tablet report...
I20260808 02:21:42.791234 23615 tablet_bootstrap.cc:492] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: No bootstrap required, opened a new log
I20260808 02:21:42.791311 23615 ts_tablet_manager.cc:1397] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:42.792693 23615 raft_consensus.cc:348] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.792851 23615 raft_consensus.cc:374] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:42.792933 23615 raft_consensus.cc:729] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503839bee78841d3999ccea15317bd96, State: Initialized, Role: FOLLOWER
I20260808 02:21:42.793056 23615 consensus_queue.cc:260] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.793342 23615 ts_tablet_manager.cc:1428] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:42.793370 23579 heartbeater.cc:503] Master 127.28.177.190:42925 was elected leader, sending a full tablet report...
W20260808 02:21:42.836083 23374 tablet.cc:2369] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:21:42.857911 23617 raft_consensus.cc:482] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:42.858239 23617 raft_consensus.cc:504] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.858697 23617 leader_election.cc:288] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a464db250934e0f9396f5b3abd537f8 (127.28.177.129:45513), 503839bee78841d3999ccea15317bd96 (127.28.177.131:36921)
I20260808 02:21:42.862964 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.859624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:40159 (local address 127.28.177.131:36921)
0808 02:21:42.859709 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.859711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.859908 (+ 197us) server_negotiation.cc:407] Connection header received
0808 02:21:42.859938 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.859940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.859949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.860007 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:42.860286 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.860982 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.862146 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.862596 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.862686 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.862695 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.862846 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.862892 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:21:42.862977 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.859470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:36921 (local address 127.28.177.130:40159)
0808 02:21:42.859787 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.859798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.859862 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.860093 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.860095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.860097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:42.860168 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.860170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.861062 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.861064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.862056 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.862059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.862685 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.862686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.862757 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.862762 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.862841 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.862884 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":176,"threads_started":1}
I20260808 02:21:42.863106 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.859308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:58747 (local address 127.28.177.129:45513)
0808 02:21:42.859421 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.859422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.859434 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:21:42.859510 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.859511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.859524 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.859682 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:42.859903 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.860730 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.862178 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.862713 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.862808 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.862818 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.862989 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.863032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:21:42.863106 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.859261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45513 (local address 127.28.177.130:58747)
0808 02:21:42.859364 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.859378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:42.859437 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:42.859720 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:42.859721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:42.859723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:42.859812 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.859815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.860872 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.860873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.862056 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:42.862059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.862805 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.862806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:42.862901 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.862904 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:42.862980 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:21:42.863025 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":1}
I20260808 02:21:42.863600 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1e6925e6d4b4a990b6fa3d6defedf" candidate_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a464db250934e0f9396f5b3abd537f8" is_pre_election: true
I20260808 02:21:42.863595 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1e6925e6d4b4a990b6fa3d6defedf" candidate_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "503839bee78841d3999ccea15317bd96" is_pre_election: true
I20260808 02:21:42.863794 23088 raft_consensus.cc:2454] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4590fbe5b2f54b8fa61d82b74bb3c8ca in term 0.
I20260808 02:21:42.863794 23508 raft_consensus.cc:2454] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4590fbe5b2f54b8fa61d82b74bb3c8ca in term 0.
I20260808 02:21:42.864082 23196 leader_election.cc:302] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a464db250934e0f9396f5b3abd537f8, 4590fbe5b2f54b8fa61d82b74bb3c8ca; no voters:
I20260808 02:21:42.864285 23617 raft_consensus.cc:2781] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:42.864377 23617 raft_consensus.cc:482] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:42.864396 23617 raft_consensus.cc:3037] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:42.865571 23617 raft_consensus.cc:504] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.865737 23617 leader_election.cc:288] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [CANDIDATE]: Term 1 election: Requested vote from peers 1a464db250934e0f9396f5b3abd537f8 (127.28.177.129:45513), 503839bee78841d3999ccea15317bd96 (127.28.177.131:36921)
I20260808 02:21:42.865998 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1e6925e6d4b4a990b6fa3d6defedf" candidate_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a464db250934e0f9396f5b3abd537f8"
I20260808 02:21:42.865998 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1e6925e6d4b4a990b6fa3d6defedf" candidate_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "503839bee78841d3999ccea15317bd96"
I20260808 02:21:42.866101 23088 raft_consensus.cc:3037] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:42.866109 23508 raft_consensus.cc:3037] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:42.867107 23088 raft_consensus.cc:2454] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4590fbe5b2f54b8fa61d82b74bb3c8ca in term 1.
I20260808 02:21:42.867183 23508 raft_consensus.cc:2454] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4590fbe5b2f54b8fa61d82b74bb3c8ca in term 1.
I20260808 02:21:42.867309 23196 leader_election.cc:302] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a464db250934e0f9396f5b3abd537f8, 4590fbe5b2f54b8fa61d82b74bb3c8ca; no voters:
I20260808 02:21:42.867432 23617 raft_consensus.cc:2781] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:42.867633 23617 raft_consensus.cc:686] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 1 LEADER]: Becoming Leader. State: Replica: 4590fbe5b2f54b8fa61d82b74bb3c8ca, State: Running, Role: LEADER
I20260808 02:21:42.867751 23617 consensus_queue.cc:237] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } }
I20260808 02:21:42.868650 22849 catalog_manager.cc:5495] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4590fbe5b2f54b8fa61d82b74bb3c8ca (127.28.177.130). New cstate: current_term: 1 leader_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4590fbe5b2f54b8fa61d82b74bb3c8ca" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36141 } health_report { overall_health: HEALTHY } } }
I20260808 02:21:42.877074 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.874468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.28.177.130:36141)
0808 02:21:42.874535 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:42.874537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:42.874545 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:21:42.874586 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:42.874587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:42.874604 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:42.874689 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:42.874823 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.875332 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.876183 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:42.876400 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:42.876475 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:42.876689 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:42.876782 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:42.876913 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:42.876970 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 02:21:42.876980 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:21:42.879218 23088 raft_consensus.cc:1261] T 50e1e6925e6d4b4a990b6fa3d6defedf P 1a464db250934e0f9396f5b3abd537f8 [term 1 FOLLOWER]: Refusing update from remote peer 4590fbe5b2f54b8fa61d82b74bb3c8ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:42.879278 23508 raft_consensus.cc:1261] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 1 FOLLOWER]: Refusing update from remote peer 4590fbe5b2f54b8fa61d82b74bb3c8ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:42.879688 23625 consensus_queue.cc:1035] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a464db250934e0f9396f5b3abd537f8" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 45513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:42.879890 23617 consensus_queue.cc:1035] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "503839bee78841d3999ccea15317bd96" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:42.883050 23632 mvcc.cc:205] Tried to move back new op lower bound from 7316093758990028800 to 7316093758946811904. Current Snapshot: MvccSnapshot[applied={T|T < 7316093758990028800 or (T in {7316093758990028800})}]
I20260808 02:21:42.883636 23633 mvcc.cc:205] Tried to move back new op lower bound from 7316093758990028800 to 7316093758946811904. Current Snapshot: MvccSnapshot[applied={T|T < 7316093758990028800}]
I20260808 02:21:42.885260 22849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:42.885366 22849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:42.886222 22849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd52de0631e7489fb50c219dda2f10d5: Sending DeleteTablet for 3 replicas of tablet 50e1e6925e6d4b4a990b6fa3d6defedf
I20260808 02:21:42.886581 23055 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e1e6925e6d4b4a990b6fa3d6defedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:42 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:51404
I20260808 02:21:42.886643 23267 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e1e6925e6d4b4a990b6fa3d6defedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:42 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:38470
I20260808 02:21:42.886749 23478 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e1e6925e6d4b4a990b6fa3d6defedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:42 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:41858
I20260808 02:21:42.886946 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 22958
I20260808 02:21:42.898488 23651 tablet_replica.cc:330] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: stopping tablet replica
I20260808 02:21:42.898669 23651 raft_consensus.cc:2229] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:42.898819 23651 raft_consensus.cc:2258] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:42.899312 23651 ts_tablet_manager.cc:1905] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:42.901087 23651 ts_tablet_manager.cc:1918] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:42.901161 23651 log.cc:1196] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/50e1e6925e6d4b4a990b6fa3d6defedf
I20260808 02:21:42.901528 23651 ts_tablet_manager.cc:1939] T 50e1e6925e6d4b4a990b6fa3d6defedf P 503839bee78841d3999ccea15317bd96: Deleting consensus metadata
I20260808 02:21:42.902084 22820 catalog_manager.cc:4841] TS 503839bee78841d3999ccea15317bd96 (127.28.177.131:36921): tablet 50e1e6925e6d4b4a990b6fa3d6defedf (table test-table [id=ec0b824d90114c16bb006eb32e922ced]) successfully deleted
I20260808 02:21:42.905339 23650 tablet_replica.cc:330] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca: stopping tablet replica
I20260808 02:21:42.905517 23650 raft_consensus.cc:2229] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:21:42.905678 23650 raft_consensus.cc:2258] T 50e1e6925e6d4b4a990b6fa3d6defedf P 4590fbe5b2f54b8fa61d82b74bb3c8ca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:21:42.906067 22820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:45513 (error 108)
I20260808 02:21:42.906301 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 23169
I20260808 02:21:42.912039 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.911844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:45513 (local address 127.0.0.1:51416)
0808 02:21:42.911929 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.911991 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:21:42.912302 22820 catalog_manager.cc:4584] TS 1a464db250934e0f9396f5b3abd537f8 (127.28.177.129:45513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50e1e6925e6d4b4a990b6fa3d6defedf: Network error: Client connection negotiation failed: client connection to 127.28.177.129:45513: connect: Connection refused (error 111)
I20260808 02:21:42.920778 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 23379
I20260808 02:21:42.921026 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:42.920778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:36141 (local address 127.0.0.1:38476)
0808 02:21:42.920914 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:42.920953 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.130:36141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:21:42.929159 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 22797
2026-08-08T02:21:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2312 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T02:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:42Z Disabled control of system clock
I20260808 02:21:42.969292 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:44537
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:44537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:43.054428 23662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:43.054724 23662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:43.054764 23662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:43.055958 23662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:43.056030 23662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:43.056052 23662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:43.056073 23662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:43.056095 23662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:43.057528 23662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:44537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:44537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:43.057806 23662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:43.058056 23662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:43.063421 23669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.063529 23673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.063433 23668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:43.063829 23662 server_base.cc:1034] running on GCE node
I20260808 02:21:43.064208 23662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:43.064524 23662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:43.065694 23662 hybrid_clock.cc:648] HybridClock initialized: now 1786155703065673 us; error 40 us; skew 500 ppm
I20260808 02:21:43.067162 23662 webserver.cc:460] Webserver started at http://127.28.177.190:46253/ using document root <none> and password file <none>
I20260808 02:21:43.067462 23662 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:43.067561 23662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:43.067749 23662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:43.069226 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "c0b7877d322d4947a53545885a8598d7"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "b0912ff527110e7893739e61c6c1c940"
server_key_iv: "0eaf4c85073ada190291366f971b26b7"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.069641 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "c0b7877d322d4947a53545885a8598d7"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "b0912ff527110e7893739e61c6c1c940"
server_key_iv: "0eaf4c85073ada190291366f971b26b7"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.071194 23662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:43.072057 23684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.072291 23662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:43.072423 23662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "c0b7877d322d4947a53545885a8598d7"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "b0912ff527110e7893739e61c6c1c940"
server_key_iv: "0eaf4c85073ada190291366f971b26b7"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.072497 23662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:43.099238 23662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:43.099671 23662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:43.099849 23662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:43.104492 23662 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:44537
I20260808 02:21:43.104558 23758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:44537 every 8 connection(s)
I20260808 02:21:43.104995 23662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:43.106163 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:43.108603 23759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: Bootstrap starting.
I20260808 02:21:43.109323 23759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:43.109649 23759 log.cc:824] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:43.110471 23759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: No bootstrap required, opened a new log
I20260808 02:21:43.112119 23759 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b7877d322d4947a53545885a8598d7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44537 } }
I20260808 02:21:43.112269 23759 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:43.112291 23759 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b7877d322d4947a53545885a8598d7, State: Initialized, Role: FOLLOWER
I20260808 02:21:43.112457 23759 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b7877d322d4947a53545885a8598d7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44537 } }
I20260808 02:21:43.112521 23759 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:43.112542 23759 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:43.112593 23759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:43.113234 23759 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b7877d322d4947a53545885a8598d7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44537 } }
I20260808 02:21:43.113349 23759 leader_election.cc:302] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0b7877d322d4947a53545885a8598d7; no voters:
I20260808 02:21:43.113557 23759 leader_election.cc:288] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:43.113617 23764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:43.113863 23764 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [term 1 LEADER]: Becoming Leader. State: Replica: c0b7877d322d4947a53545885a8598d7, State: Running, Role: LEADER
I20260808 02:21:43.113897 23759 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:43.113979 23764 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b7877d322d4947a53545885a8598d7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44537 } }
I20260808 02:21:43.114580 23767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0b7877d322d4947a53545885a8598d7. Latest consensus state: current_term: 1 leader_uuid: "c0b7877d322d4947a53545885a8598d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b7877d322d4947a53545885a8598d7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44537 } } }
I20260808 02:21:43.114643 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 23662
I20260808 02:21:43.114758 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:43.114997 29382 external_mini_cluster.cc:1428] Setting key 9abb05df0d3b2452b959b44becebe36a
I20260808 02:21:43.115087 23765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0b7877d322d4947a53545885a8598d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b7877d322d4947a53545885a8598d7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44537 } } }
I20260808 02:21:43.115211 23765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:43.115190 23767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7 [sys.catalog]: This master's current role is: LEADER
W20260808 02:21:43.115972 23791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:21:43.116034 23791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:21:43.116158 23792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:43.116494 23792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:43.118191 23792 catalog_manager.cc:1347] Generated new cluster ID: 0ba0cd82eb4d4a3f90a95aa470a967c5
I20260808 02:21:43.118299 23792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:43.120468 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.115611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.28.177.190:44537)
0808 02:21:43.115806 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.115817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.115856 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:21:43.115999 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.116005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.116104 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.116299 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:43.116543 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.117740 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.118539 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.118956 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.119093 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.119807 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:43.119810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:43.119811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:43.119985 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:43.120017 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:43.120028 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:43.120108 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.120190 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.120271 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260808 02:21:43.136104 23792 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:43.136663 23792 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:43.140257 23792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: Generated new TSK 0
I20260808 02:21:43.140468 23792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:43.149107 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44537
--builtin_ntp_servers=127.28.177.148:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:43.239926 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:43.240213 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:43.240257 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:43.241655 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:43.241726 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:43.241770 23800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:43.243369 23800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:43.243691 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:43.244082 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:43.249912 23812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.249876 23811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.249969 23814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:43.250339 23800 server_base.cc:1034] running on GCE node
I20260808 02:21:43.250562 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:43.250833 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:43.251995 23800 hybrid_clock.cc:648] HybridClock initialized: now 1786155703251980 us; error 31 us; skew 500 ppm
I20260808 02:21:43.253329 23800 webserver.cc:460] Webserver started at http://127.28.177.129:36007/ using document root <none> and password file <none>
I20260808 02:21:43.253592 23800 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:43.253702 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:43.253860 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:43.254998 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "8c7562b7ef43427e8c51a1a92269d9f1"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "6abd69152ea97868baec009dd6317ae9"
server_key_iv: "5ac18978849b2f7fc3e881b7c2175538"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.255369 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "8c7562b7ef43427e8c51a1a92269d9f1"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "6abd69152ea97868baec009dd6317ae9"
server_key_iv: "5ac18978849b2f7fc3e881b7c2175538"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.256935 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:43.257856 23821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.258081 23800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:43.258206 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "8c7562b7ef43427e8c51a1a92269d9f1"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "6abd69152ea97868baec009dd6317ae9"
server_key_iv: "5ac18978849b2f7fc3e881b7c2175538"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.258298 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:43.281124 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:43.281493 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:43.281665 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:43.281926 23800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:43.282312 23800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:43.282379 23800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.282414 23800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:43.282445 23800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.288486 23800 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:35019
I20260808 02:21:43.288604 24007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:35019 every 8 connection(s)
I20260808 02:21:43.288996 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:43.294835 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.289854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:51545 (local address 127.28.177.190:44537)
0808 02:21:43.289927 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.289928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.290216 (+ 288us) server_negotiation.cc:407] Connection header received
0808 02:21:43.290793 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.290794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.290811 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.290864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:43.291583 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.292116 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.293227 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.293436 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.293524 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.294456 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:43.294457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:43.294458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:43.294464 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:43.294495 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:43.294500 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:43.294578 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.294699 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.294741 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:21:43.295100 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.289828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:44537 (local address 127.28.177.129:51545)
0808 02:21:43.290052 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.290087 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.290659 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.290953 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.290957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.290966 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:43.291485 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.291490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.292225 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.292226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.293099 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.293101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.293540 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.293541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.293696 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.294238 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:43.294238 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:43.294272 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:43.294297 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:43.294596 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:43.294603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:43.294607 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.294702 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.294822 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260808 02:21:43.295840 24009 heartbeater.cc:347] Connected to a master server at 127.28.177.190:44537
I20260808 02:21:43.295959 24009 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:43.295961 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 23800
I20260808 02:21:43.296224 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:21:43.296314 24009 heartbeater.cc:511] Master 127.28.177.190:44537 requested a full tablet report, sending...
I20260808 02:21:43.296497 29382 external_mini_cluster.cc:1428] Setting key 4097433f0483524290c62ab7fc1b50c3
I20260808 02:21:43.297134 23707 ts_manager.cc:194] Registered new tserver with Master: 8c7562b7ef43427e8c51a1a92269d9f1 (127.28.177.129:35019)
I20260808 02:21:43.298123 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44537
--builtin_ntp_servers=127.28.177.148:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:21:43.298133 23707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.129:51545
W20260808 02:21:43.387333 24013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:43.387602 24013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:43.387629 24013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:43.388909 24013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:43.388953 24013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:43.388981 24013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:43.390395 24013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:43.390681 24013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:43.390997 24013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:43.396584 24021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.396621 24018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.396649 24019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:43.396874 24013 server_base.cc:1034] running on GCE node
I20260808 02:21:43.397130 24013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:43.397419 24013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:43.398582 24013 hybrid_clock.cc:648] HybridClock initialized: now 1786155703398543 us; error 49 us; skew 500 ppm
I20260808 02:21:43.399945 24013 webserver.cc:460] Webserver started at http://127.28.177.130:33709/ using document root <none> and password file <none>
I20260808 02:21:43.400206 24013 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:43.400280 24013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:43.400432 24013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:43.401605 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "ec570b02d20949d7b92d3d7a4cbb080a"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "06d105696132719e2b0e78021d109b04"
server_key_iv: "90f101c4241df7027ed57a5530765644"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.401978 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "ec570b02d20949d7b92d3d7a4cbb080a"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "06d105696132719e2b0e78021d109b04"
server_key_iv: "90f101c4241df7027ed57a5530765644"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.403384 24013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:43.404314 24032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.404616 24013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:43.404738 24013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "ec570b02d20949d7b92d3d7a4cbb080a"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "06d105696132719e2b0e78021d109b04"
server_key_iv: "90f101c4241df7027ed57a5530765644"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.404819 24013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:43.428491 24013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:43.428937 24013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:43.429131 24013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:43.429425 24013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:43.429797 24013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:43.429854 24013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.429901 24013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:43.429926 24013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.435943 24013 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:36789
I20260808 02:21:43.435990 24212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:36789 every 8 connection(s)
I20260808 02:21:43.436381 24013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:21:43.442241 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.437257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:57829 (local address 127.28.177.190:44537)
0808 02:21:43.437343 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.437345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.437605 (+ 260us) server_negotiation.cc:407] Connection header received
0808 02:21:43.438396 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.438397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.438413 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.438489 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:43.439238 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.439780 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.440782 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.441002 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.441133 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.441955 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:43.441956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:43.441956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:43.441960 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:43.441973 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:43.441977 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:43.442024 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.442146 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.442172 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:21:43.442375 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.437273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:44537 (local address 127.28.177.130:57829)
0808 02:21:43.437457 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.437493 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.438037 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.438560 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.438563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.438572 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:43.439116 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.439121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.439880 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.439881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.440689 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.440692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.441176 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.441177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.441292 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.441820 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:43.441820 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:43.441849 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:43.441872 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:43.442024 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:43.442027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:43.442029 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.442078 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.442170 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260808 02:21:43.443037 24213 heartbeater.cc:347] Connected to a master server at 127.28.177.190:44537
I20260808 02:21:43.443173 24213 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:43.443401 24213 heartbeater.cc:511] Master 127.28.177.190:44537 requested a full tablet report, sending...
I20260808 02:21:43.443882 23707 ts_manager.cc:194] Registered new tserver with Master: ec570b02d20949d7b92d3d7a4cbb080a (127.28.177.130:36789)
I20260808 02:21:43.444353 23707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.130:57829
I20260808 02:21:43.446211 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 24013
I20260808 02:21:43.446319 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:21:43.446556 29382 external_mini_cluster.cc:1428] Setting key 2cfb2f434b185bb401245228373ab12e
I20260808 02:21:43.447674 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44537
--builtin_ntp_servers=127.28.177.148:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:43.545804 24218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:43.546074 24218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:43.546113 24218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:43.547555 24218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:43.547618 24218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:43.547652 24218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:43.549335 24218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:44537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:43.549660 24218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:43.550045 24218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:43.556272 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.556344 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:43.556349 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:43.556418 24218 server_base.cc:1034] running on GCE node
I20260808 02:21:43.556764 24218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:43.557139 24218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:43.558337 24218 hybrid_clock.cc:648] HybridClock initialized: now 1786155703558311 us; error 46 us; skew 500 ppm
I20260808 02:21:43.559990 24218 webserver.cc:460] Webserver started at http://127.28.177.131:37027/ using document root <none> and password file <none>
I20260808 02:21:43.560333 24218 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:43.560390 24218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:43.560535 24218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:43.561852 24218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "54384ec6db31490a96f14d25cad582a4"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "a3fc8cfcc07f1af34b11334c23f1bd18"
server_key_iv: "04e88e62c8f1241d5407b783f1d2dcd4"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.562249 24218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "54384ec6db31490a96f14d25cad582a4"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "a3fc8cfcc07f1af34b11334c23f1bd18"
server_key_iv: "04e88e62c8f1241d5407b783f1d2dcd4"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.563722 24218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:21:43.564550 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.564749 24218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:43.564885 24218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "54384ec6db31490a96f14d25cad582a4"
format_stamp: "Formatted at 2026-08-08 02:21:43 on dist-test-slave-3h5h"
server_key: "a3fc8cfcc07f1af34b11334c23f1bd18"
server_key_iv: "04e88e62c8f1241d5407b783f1d2dcd4"
server_key_version: "encryptionkey@0"
I20260808 02:21:43.564973 24218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:43.571810 24218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:43.572134 24218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:43.572300 24218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:43.572527 24218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:43.572938 24218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:43.572994 24218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.573038 24218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:43.573069 24218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:43.580417 24218 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:37481
I20260808 02:21:43.580461 24408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:37481 every 8 connection(s)
I20260808 02:21:43.580953 24218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:43.584218 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 24218
I20260808 02:21:43.584349 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:21:43.584586 29382 external_mini_cluster.cc:1428] Setting key 89d6a6d6ea5530d9613b196609db9732
I20260808 02:21:43.587384 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.581865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:46025 (local address 127.28.177.190:44537)
0808 02:21:43.581946 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.581948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.582205 (+ 257us) server_negotiation.cc:407] Connection header received
0808 02:21:43.582680 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.582680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.582692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.582768 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:43.583514 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.584067 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.585714 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.585930 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.586023 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.586986 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:43.586988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:43.586988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:43.586993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:43.587016 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:43.587020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:43.587160 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.587251 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.587290 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:21:43.587682 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.581855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:44537 (local address 127.28.177.131:46025)
0808 02:21:43.582045 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.582088 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.582592 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.582824 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.582827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.582835 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:43.583379 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.583384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.584216 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.584217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.585605 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.585608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.586039 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.586040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.586195 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.586718 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:43.586718 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:43.586748 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:43.586773 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:43.587164 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:43.587173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:43.587176 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.587254 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.587391 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260808 02:21:43.588207 24410 heartbeater.cc:347] Connected to a master server at 127.28.177.190:44537
I20260808 02:21:43.588327 24410 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:43.588577 24410 heartbeater.cc:511] Master 127.28.177.190:44537 requested a full tablet report, sending...
I20260808 02:21:43.589212 23707 ts_manager.cc:194] Registered new tserver with Master: 54384ec6db31490a96f14d25cad582a4 (127.28.177.131:37481)
I20260808 02:21:43.589643 23707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.131:46025
I20260808 02:21:43.596781 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:43.601373 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.598406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.28.177.190:44537)
0808 02:21:43.598481 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.598483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.598595 (+ 112us) server_negotiation.cc:407] Connection header received
0808 02:21:43.598634 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.598635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.598647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.598696 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:43.598844 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.599393 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.600119 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.600316 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.600368 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.601126 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:43.601127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:43.601127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:43.601132 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:43.601145 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:43.601149 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:43.601232 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.601302 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.601313 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:21:43.603200 23707 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:43.604980 23707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:43.612071 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.607600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.28.177.129:35019)
0808 02:21:43.608134 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.608138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.608148 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:43.608191 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.608192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.608207 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.608284 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:43.608524 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.609722 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.610993 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.611636 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.611737 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.611839 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.611930 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.611978 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":149,"threads_started":1}
I20260808 02:21:43.612069 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:37481 (local address 127.0.0.1:37198)
0808 02:21:43.607622 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.607664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.607730 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.608288 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.608291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.608294 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:43.608377 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.608380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.609508 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.609509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.610807 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.610809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.611805 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.611806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.611875 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.611880 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.611914 (+ 34us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.611995 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":56,"thread_start_us":78,"threads_started":1}
I20260808 02:21:43.612224 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.607892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.28.177.131:37481)
0808 02:21:43.608074 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.608079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.608092 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:21:43.608144 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.608146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.608163 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.608458 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:43.608519 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.609421 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.610909 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.611699 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.611979 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.612073 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.612098 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.612138 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260808 02:21:43.612269 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.607807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.28.177.130:36789)
0808 02:21:43.608028 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.608033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.608306 (+ 273us) server_negotiation.cc:407] Connection header received
0808 02:21:43.608350 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.608352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.608368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.608440 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:43.608633 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.609768 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.610946 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.611699 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.611777 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.611872 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.612133 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.612181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260808 02:21:43.612649 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:35019 (local address 127.0.0.1:48674)
0808 02:21:43.607857 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.607868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.607935 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.608375 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.608376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.608378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:43.608427 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.608429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.609816 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.609817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.610883 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.610885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.611737 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.611738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.611817 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.611837 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.611914 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.612567 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":67,"threads_started":1}
I20260808 02:21:43.612810 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.607687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:36789 (local address 127.0.0.1:51696)
0808 02:21:43.608181 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.608194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.608243 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.608507 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.608508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.608510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:43.608544 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.608545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.609865 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.609866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.610828 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.610830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.611906 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.611906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.611982 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.611985 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.612045 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.612089 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260808 02:21:43.613137 24312 tablet_service.cc:1467] Processing CreateTablet for tablet 69a8c4f15224405ab5407c1ae298891a (DEFAULT_TABLE table=test-table [id=1f9866e8e5e44d48ae1c6d6293ad6faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:43.613559 24312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a8c4f15224405ab5407c1ae298891a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:43.613485 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 69a8c4f15224405ab5407c1ae298891a (DEFAULT_TABLE table=test-table [id=1f9866e8e5e44d48ae1c6d6293ad6faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:43.613865 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a8c4f15224405ab5407c1ae298891a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:43.617413 24430 tablet_bootstrap.cc:492] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: Bootstrap starting.
I20260808 02:21:43.617538 24432 tablet_bootstrap.cc:492] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Bootstrap starting.
I20260808 02:21:43.618031 24112 tablet_service.cc:1467] Processing CreateTablet for tablet 69a8c4f15224405ab5407c1ae298891a (DEFAULT_TABLE table=test-table [id=1f9866e8e5e44d48ae1c6d6293ad6faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:43.618172 24430 tablet_bootstrap.cc:654] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:43.618207 24432 tablet_bootstrap.cc:654] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:43.618389 24112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a8c4f15224405ab5407c1ae298891a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:43.618533 24432 log.cc:824] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:43.618535 24430 log.cc:824] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:43.619333 24432 tablet_bootstrap.cc:492] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: No bootstrap required, opened a new log
I20260808 02:21:43.619412 24432 ts_tablet_manager.cc:1397] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:21:43.619571 24430 tablet_bootstrap.cc:492] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: No bootstrap required, opened a new log
I20260808 02:21:43.619666 24430 ts_tablet_manager.cc:1397] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:21:43.622303 24432 raft_consensus.cc:348] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.622308 24430 raft_consensus.cc:348] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.622476 24430 raft_consensus.cc:374] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:43.622476 24432 raft_consensus.cc:374] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:43.622514 24430 raft_consensus.cc:729] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7562b7ef43427e8c51a1a92269d9f1, State: Initialized, Role: FOLLOWER
I20260808 02:21:43.622514 24432 raft_consensus.cc:729] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54384ec6db31490a96f14d25cad582a4, State: Initialized, Role: FOLLOWER
I20260808 02:21:43.622653 24430 consensus_queue.cc:260] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.622654 24432 consensus_queue.cc:260] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.622928 24430 ts_tablet_manager.cc:1428] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:21:43.622984 24410 heartbeater.cc:503] Master 127.28.177.190:44537 was elected leader, sending a full tablet report...
I20260808 02:21:43.623077 24009 heartbeater.cc:503] Master 127.28.177.190:44537 was elected leader, sending a full tablet report...
I20260808 02:21:43.622928 24432 ts_tablet_manager.cc:1428] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 02:21:43.623629 24438 tablet_bootstrap.cc:492] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Bootstrap starting.
I20260808 02:21:43.624459 24438 tablet_bootstrap.cc:654] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:43.624835 24438 log.cc:824] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:43.625648 24438 tablet_bootstrap.cc:492] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: No bootstrap required, opened a new log
I20260808 02:21:43.625706 24438 ts_tablet_manager.cc:1397] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:21:43.627540 24438 raft_consensus.cc:348] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.627684 24438 raft_consensus.cc:374] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:43.627707 24438 raft_consensus.cc:729] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec570b02d20949d7b92d3d7a4cbb080a, State: Initialized, Role: FOLLOWER
I20260808 02:21:43.627852 24438 consensus_queue.cc:260] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.628113 24438 ts_tablet_manager.cc:1428] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:21:43.628165 24213 heartbeater.cc:503] Master 127.28.177.190:44537 was elected leader, sending a full tablet report...
W20260808 02:21:43.687106 24215 tablet.cc:2369] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:21:43.789846 24010 tablet.cc:2369] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:21:43.831653 24412 tablet.cc:2369] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:21:43.944820 24444 raft_consensus.cc:482] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:43.945035 24444 raft_consensus.cc:504] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.945462 24444 leader_election.cc:288] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c7562b7ef43427e8c51a1a92269d9f1 (127.28.177.129:35019), ec570b02d20949d7b92d3d7a4cbb080a (127.28.177.130:36789)
I20260808 02:21:43.948555 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.945796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:37875 (local address 127.28.177.129:35019)
0808 02:21:43.945846 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.945849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.945861 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:21:43.945927 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.945928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.945943 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.946118 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:43.946297 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.946795 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.947772 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.948176 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.948267 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.948277 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.948437 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.948476 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:21:43.948565 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.945687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:35019 (local address 127.28.177.131:37875)
0808 02:21:43.945766 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.945781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.945854 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.946141 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.946142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.946144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:43.946220 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.946222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.946908 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.946909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.947676 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.947678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.948269 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.948270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.948348 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.948351 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.948427 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.948481 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 02:21:43.948567 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.945796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:41407 (local address 127.28.177.130:36789)
0808 02:21:43.945839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.945841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.946021 (+ 180us) server_negotiation.cc:407] Connection header received
0808 02:21:43.946158 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.946160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.946173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.946252 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:43.946415 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.946886 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.947833 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.948194 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.948281 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.948289 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.948449 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.948489 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:21:43.948580 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.945687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:36789 (local address 127.28.177.131:41407)
0808 02:21:43.945936 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.945945 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:43.946026 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:43.946256 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:43.946257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:43.946258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:43.946339 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.946341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.946983 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.946984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.947733 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:43.947735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.948281 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.948282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:43.948368 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.948371 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:43.948442 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:21:43.948482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":75,"thread_start_us":110,"threads_started":1}
I20260808 02:21:43.948957 23933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a8c4f15224405ab5407c1ae298891a" candidate_uuid: "54384ec6db31490a96f14d25cad582a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" is_pre_election: true
I20260808 02:21:43.949131 23933 raft_consensus.cc:2454] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54384ec6db31490a96f14d25cad582a4 in term 0.
I20260808 02:21:43.949194 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a8c4f15224405ab5407c1ae298891a" candidate_uuid: "54384ec6db31490a96f14d25cad582a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" is_pre_election: true
I20260808 02:21:43.949370 24145 raft_consensus.cc:2454] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54384ec6db31490a96f14d25cad582a4 in term 0.
I20260808 02:21:43.949448 24244 leader_election.cc:302] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54384ec6db31490a96f14d25cad582a4, 8c7562b7ef43427e8c51a1a92269d9f1; no voters:
I20260808 02:21:43.949584 24444 raft_consensus.cc:2781] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:43.949636 24444 raft_consensus.cc:482] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:43.949667 24444 raft_consensus.cc:3037] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:43.950405 24444 raft_consensus.cc:504] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.950567 24444 leader_election.cc:288] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [CANDIDATE]: Term 1 election: Requested vote from peers 8c7562b7ef43427e8c51a1a92269d9f1 (127.28.177.129:35019), ec570b02d20949d7b92d3d7a4cbb080a (127.28.177.130:36789)
I20260808 02:21:43.950688 23933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a8c4f15224405ab5407c1ae298891a" candidate_uuid: "54384ec6db31490a96f14d25cad582a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7562b7ef43427e8c51a1a92269d9f1"
I20260808 02:21:43.950788 23933 raft_consensus.cc:3037] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:43.950843 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a8c4f15224405ab5407c1ae298891a" candidate_uuid: "54384ec6db31490a96f14d25cad582a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec570b02d20949d7b92d3d7a4cbb080a"
I20260808 02:21:43.950973 24145 raft_consensus.cc:3037] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:43.951583 23933 raft_consensus.cc:2454] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54384ec6db31490a96f14d25cad582a4 in term 1.
I20260808 02:21:43.951751 24244 leader_election.cc:302] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54384ec6db31490a96f14d25cad582a4, 8c7562b7ef43427e8c51a1a92269d9f1; no voters:
I20260808 02:21:43.951885 24444 raft_consensus.cc:2781] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:43.951965 24145 raft_consensus.cc:2454] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54384ec6db31490a96f14d25cad582a4 in term 1.
I20260808 02:21:43.952111 24444 raft_consensus.cc:686] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 1 LEADER]: Becoming Leader. State: Replica: 54384ec6db31490a96f14d25cad582a4, State: Running, Role: LEADER
I20260808 02:21:43.952338 24444 consensus_queue.cc:237] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } }
I20260808 02:21:43.953408 23706 catalog_manager.cc:5495] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54384ec6db31490a96f14d25cad582a4 (127.28.177.131). New cstate: current_term: 1 leader_uuid: "54384ec6db31490a96f14d25cad582a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54384ec6db31490a96f14d25cad582a4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 } health_report { overall_health: UNKNOWN } } }
I20260808 02:21:43.967531 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.965006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.28.177.131:37481)
0808 02:21:43.965073 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:43.965076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:43.965087 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:21:43.965159 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:43.965160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:43.965181 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:43.965258 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:43.965403 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.965877 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.966687 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:43.966886 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:43.966992 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:43.967156 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:43.967266 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:43.967380 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:43.967411 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 02:21:43.967419 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:21:43.969681 23933 raft_consensus.cc:1261] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 1 FOLLOWER]: Refusing update from remote peer 54384ec6db31490a96f14d25cad582a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:43.969777 24145 raft_consensus.cc:1261] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 1 FOLLOWER]: Refusing update from remote peer 54384ec6db31490a96f14d25cad582a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:43.970139 24444 consensus_queue.cc:1035] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7562b7ef43427e8c51a1a92269d9f1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:43.970402 24444 consensus_queue.cc:1035] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec570b02d20949d7b92d3d7a4cbb080a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:43.972728 24454 mvcc.cc:205] Tried to move back new op lower bound from 7316093763456299008 to 7316093763389509632. Current Snapshot: MvccSnapshot[applied={T|T < 7316093763456299008}]
I20260808 02:21:43.973001 24456 mvcc.cc:205] Tried to move back new op lower bound from 7316093763456299008 to 7316093763389509632. Current Snapshot: MvccSnapshot[applied={T|T < 7316093763456299008}]
I20260808 02:21:43.973229 24457 mvcc.cc:205] Tried to move back new op lower bound from 7316093763456299008 to 7316093763389509632. Current Snapshot: MvccSnapshot[applied={T|T < 7316093763456299008}]
I20260808 02:21:43.976020 23707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:43.976127 23707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:43.977072 23707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0b7877d322d4947a53545885a8598d7: Sending DeleteTablet for 3 replicas of tablet 69a8c4f15224405ab5407c1ae298891a
I20260808 02:21:43.977421 24312 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a8c4f15224405ab5407c1ae298891a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:43 UTC) from {username='slave'} at 127.0.0.1:37198
I20260808 02:21:43.977450 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a8c4f15224405ab5407c1ae298891a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:43 UTC) from {username='slave'} at 127.0.0.1:48674
I20260808 02:21:43.977600 24112 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a8c4f15224405ab5407c1ae298891a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:43 UTC) from {username='slave'} at 127.0.0.1:51696
I20260808 02:21:43.977684 24467 tablet_replica.cc:330] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: stopping tablet replica
I20260808 02:21:43.977700 24468 tablet_replica.cc:330] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1: stopping tablet replica
I20260808 02:21:43.977787 24467 raft_consensus.cc:2229] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:21:43.977799 24468 raft_consensus.cc:2229] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:43.977900 24467 raft_consensus.cc:2258] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:43.977912 24468 raft_consensus.cc:2258] T 69a8c4f15224405ab5407c1ae298891a P 8c7562b7ef43427e8c51a1a92269d9f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:43.978103 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 23800
I20260808 02:21:43.978329 24467 ts_tablet_manager.cc:1905] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:43.982386 24467 ts_tablet_manager.cc:1918] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:43.982457 24467 log.cc:1196] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/69a8c4f15224405ab5407c1ae298891a
I20260808 02:21:43.982688 24467 ts_tablet_manager.cc:1939] T 69a8c4f15224405ab5407c1ae298891a P 54384ec6db31490a96f14d25cad582a4: Deleting consensus metadata
I20260808 02:21:43.983088 23688 catalog_manager.cc:4841] TS 54384ec6db31490a96f14d25cad582a4 (127.28.177.131:37481): tablet 69a8c4f15224405ab5407c1ae298891a (table test-table [id=1f9866e8e5e44d48ae1c6d6293ad6faa]) successfully deleted
I20260808 02:21:43.990103 24471 tablet_replica.cc:330] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: stopping tablet replica
I20260808 02:21:43.990278 24471 raft_consensus.cc:2229] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:43.990432 24471 raft_consensus.cc:2258] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:43.991012 24471 ts_tablet_manager.cc:1905] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:43.993338 24471 ts_tablet_manager.cc:1918] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:43.993427 24471 log.cc:1196] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/69a8c4f15224405ab5407c1ae298891a
I20260808 02:21:43.993728 24471 ts_tablet_manager.cc:1939] T 69a8c4f15224405ab5407c1ae298891a P ec570b02d20949d7b92d3d7a4cbb080a: Deleting consensus metadata
I20260808 02:21:43.994310 23688 catalog_manager.cc:4841] TS ec570b02d20949d7b92d3d7a4cbb080a (127.28.177.130:36789): tablet 69a8c4f15224405ab5407c1ae298891a (table test-table [id=1f9866e8e5e44d48ae1c6d6293ad6faa]) successfully deleted
W20260808 02:21:43.994974 23689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:35019 (error 108)
I20260808 02:21:43.995580 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:43.995419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:35019 (local address 127.0.0.1:48684)
0808 02:21:43.995483 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:43.995541 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:35019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 02:21:43.995667 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 24013
W20260808 02:21:43.995747 23689 catalog_manager.cc:4584] TS 8c7562b7ef43427e8c51a1a92269d9f1 (127.28.177.129:35019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69a8c4f15224405ab5407c1ae298891a: Network error: Client connection negotiation failed: client connection to 127.28.177.129:35019: connect: Connection refused (error 111)
I20260808 02:21:44.015568 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 24218
I20260808 02:21:44.025713 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 23662
2026-08-08T02:21:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1096 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:44 dist-test-slave-3h5h krb5kdc[24476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:44 dist-test-slave-3h5h krb5kdc[24476](info): set up 2 sockets
Aug 08 02:21:44 dist-test-slave-3h5h krb5kdc[24476](info): commencing operation
krb5kdc: starting...
W20260808 02:21:45.362355 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:45 dist-test-slave-3h5h krb5kdc[24476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:21:45.505021 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42689
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:45.596935 24492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:45.597198 24492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:45.597240 24492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:45.598429 24492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:45.598591 24492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:45.598647 24492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:45.598672 24492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:45.598701 24492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:45.600310 24492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:45.600622 24492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:45.600885 24492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:45.606206 24505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:45.606216 24501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:45.606308 24492 server_base.cc:1034] running on GCE node
W20260808 02:21:45.606206 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:45.606654 24492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:45.607039 24492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:45.608239 24492 hybrid_clock.cc:648] HybridClock initialized: now 1786155705608218 us; error 39 us; skew 500 ppm
Aug 08 02:21:45 dist-test-slave-3h5h krb5kdc[24476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:45.611243 24492 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:21:45.611754 24492 webserver.cc:460] Webserver started at http://127.28.177.190:46223/ using document root <none> and password file <none>
I20260808 02:21:45.611999 24492 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:45.612047 24492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:45.612142 24492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:45.613345 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "4e458267352845b590ddc72da038fbe8"
format_stamp: "Formatted at 2026-08-08 02:21:45 on dist-test-slave-3h5h"
server_key: "7a20eca1aeaf879986cfc8bce7ff7af9"
server_key_iv: "61e63f64503901ad6dd3e9f7da12fc7d"
server_key_version: "encryptionkey@0"
I20260808 02:21:45.613694 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "4e458267352845b590ddc72da038fbe8"
format_stamp: "Formatted at 2026-08-08 02:21:45 on dist-test-slave-3h5h"
server_key: "7a20eca1aeaf879986cfc8bce7ff7af9"
server_key_iv: "61e63f64503901ad6dd3e9f7da12fc7d"
server_key_version: "encryptionkey@0"
I20260808 02:21:45.615057 24492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:45.615814 24514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:45.616031 24492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:45.616120 24492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "4e458267352845b590ddc72da038fbe8"
format_stamp: "Formatted at 2026-08-08 02:21:45 on dist-test-slave-3h5h"
server_key: "7a20eca1aeaf879986cfc8bce7ff7af9"
server_key_iv: "61e63f64503901ad6dd3e9f7da12fc7d"
server_key_version: "encryptionkey@0"
I20260808 02:21:45.616216 24492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:45.635736 24492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:45.638046 24492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:45.638171 24492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:45.642547 24492 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42689
I20260808 02:21:45.642618 24586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42689 every 8 connection(s)
I20260808 02:21:45.643002 24492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:45.644110 24587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:45.646901 24587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8: Bootstrap starting.
I20260808 02:21:45.647609 24587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:45.647879 24587 log.cc:824] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:45.648684 24587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8: No bootstrap required, opened a new log
I20260808 02:21:45.650498 24587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e458267352845b590ddc72da038fbe8" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42689 } }
I20260808 02:21:45.650660 24587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:45.650691 24587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e458267352845b590ddc72da038fbe8, State: Initialized, Role: FOLLOWER
I20260808 02:21:45.650849 24587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e458267352845b590ddc72da038fbe8" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42689 } }
I20260808 02:21:45.650942 24587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:45.650995 24587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:45.651034 24587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:45.651592 24587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e458267352845b590ddc72da038fbe8" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42689 } }
I20260808 02:21:45.651683 24587 leader_election.cc:302] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e458267352845b590ddc72da038fbe8; no voters:
I20260808 02:21:45.651832 24587 leader_election.cc:288] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:45.651912 24594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:45.652107 24594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e458267352845b590ddc72da038fbe8, State: Running, Role: LEADER
I20260808 02:21:45.652149 24587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:45.652208 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 24492
I20260808 02:21:45.652205 24594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e458267352845b590ddc72da038fbe8" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42689 } }
I20260808 02:21:45.652348 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:45.652591 29382 external_mini_cluster.cc:1428] Setting key 500ac68b8485adb3ace5e296cdd550d3
I20260808 02:21:45.652577 24594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e458267352845b590ddc72da038fbe8. Latest consensus state: current_term: 1 leader_uuid: "4e458267352845b590ddc72da038fbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e458267352845b590ddc72da038fbe8" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42689 } } }
I20260808 02:21:45.652714 24594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:45.652745 24595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e458267352845b590ddc72da038fbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e458267352845b590ddc72da038fbe8" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42689 } } }
I20260808 02:21:45.652805 24595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:45.653151 24603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:45.654884 24603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:45.657249 24603 catalog_manager.cc:1347] Generated new cluster ID: f8670f632e444076851b9814e65187e5
I20260808 02:21:45.657308 24603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:21:45 dist-test-slave-3h5h krb5kdc[24476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:45.665541 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:45.653386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.28.177.190:42689)
0808 02:21:45.653954 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:45.653966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:45.654005 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:21:45.654147 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:45.654156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:45.654225 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:45.654492 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:45.655308 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:45.656690 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:45.657726 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:45.658235 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:45.658364 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:45.661048 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:45.661053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:45.661054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:45.661071 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:45.663655 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:45.664112 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:45.664116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:45.664116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:45.664235 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:45.664484 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:45.664486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:45.664486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:45.664772 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:45.664983 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:45.665112 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:21:45.665279 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":159,"threads_started":1}
I20260808 02:21:45.666615 24603 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:45.667322 24603 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:45.671249 24603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8: Generated new TSK 0
I20260808 02:21:45.671432 24603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.177.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
Entry for principal kudu/127.28.177.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab.
I20260808 02:21:45.704784 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42689
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:45.802911 24625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:45.803158 24625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:45.803203 24625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:45.804394 24625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:45.804477 24625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:45.804527 24625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:45.806176 24625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.129.keytab
--principal=kudu/127.28.177.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:45.806463 24625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:45.806715 24625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:45.812179 24632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:45.812244 24635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:45.812258 24637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:45.812644 24625 server_base.cc:1034] running on GCE node
I20260808 02:21:45.812924 24625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:45.813205 24625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:45.814375 24625 hybrid_clock.cc:648] HybridClock initialized: now 1786155705814350 us; error 41 us; skew 500 ppm
Aug 08 02:21:45 dist-test-slave-3h5h krb5kdc[24476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:45.817363 24625 init.cc:377] Logged in from keytab as kudu/127.28.177.129@KRBTEST.COM (short username kudu)
I20260808 02:21:45.817924 24625 webserver.cc:460] Webserver started at http://127.28.177.129:34261/ using document root <none> and password file <none>
I20260808 02:21:45.818222 24625 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:45.818284 24625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:45.818424 24625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:45.819909 24625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "0ba88f3abfdd498d9b33d6010ff41364"
format_stamp: "Formatted at 2026-08-08 02:21:45 on dist-test-slave-3h5h"
server_key: "8b9ce5dfea7e50dc08939893dbf878ad"
server_key_iv: "775f998ebb51afd7a7288515d14a5b23"
server_key_version: "encryptionkey@0"
I20260808 02:21:45.820345 24625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba88f3abfdd498d9b33d6010ff41364"
format_stamp: "Formatted at 2026-08-08 02:21:45 on dist-test-slave-3h5h"
server_key: "8b9ce5dfea7e50dc08939893dbf878ad"
server_key_iv: "775f998ebb51afd7a7288515d14a5b23"
server_key_version: "encryptionkey@0"
I20260808 02:21:45.822289 24625 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 02:21:45.823264 24646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:45.823515 24625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:45.823634 24625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "0ba88f3abfdd498d9b33d6010ff41364"
format_stamp: "Formatted at 2026-08-08 02:21:45 on dist-test-slave-3h5h"
server_key: "8b9ce5dfea7e50dc08939893dbf878ad"
server_key_iv: "775f998ebb51afd7a7288515d14a5b23"
server_key_version: "encryptionkey@0"
I20260808 02:21:45.823716 24625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:45.859620 24625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:45.862135 24625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:45.862324 24625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:45.862541 24625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:45.862818 24625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:45.862851 24625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:45.862875 24625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:45.862888 24625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:45.869571 24625 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:39465
I20260808 02:21:45.869616 24810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:39465 every 8 connection(s)
I20260808 02:21:45.869971 24625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:45.873028 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 24625
I20260808 02:21:45.873152 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:21:45.873399 29382 external_mini_cluster.cc:1428] Setting key a1b6cff5c0547af622b9b2b9f1d25287
Aug 08 02:21:45 dist-test-slave-3h5h krb5kdc[24476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.129@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:45.880573 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:45.870784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:41209 (local address 127.28.177.190:42689)
0808 02:21:45.870855 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:45.870857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:45.871073 (+ 216us) server_negotiation.cc:407] Connection header received
0808 02:21:45.871564 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:45.871565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:45.871579 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:45.871650 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:45.872619 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:45.873401 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:45.874769 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:45.875105 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:45.875192 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:45.877518 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:45.877521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:45.877521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:45.877529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:45.879028 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:45.879513 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:45.879515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:45.879515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:45.879580 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:45.879859 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:45.879860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:45.879860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:45.879984 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:45.880059 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:45.880382 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 02:21:45.880475 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:21:45.880805 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:45.870737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42689 (local address 127.28.177.129:41209)
0808 02:21:45.870928 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:45.870974 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:45.871476 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:45.871709 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:45.871712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:45.872322 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:45.872527 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:45.872532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:45.873517 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:45.873518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:45.874672 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:45.874675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:45.875206 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:45.875207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:45.875307 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:45.875867 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:45.875876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:45.877408 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:45.879137 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:45.879147 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:45.879149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:45.879430 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:45.879653 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:45.879654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:45.879654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:45.879783 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:45.880063 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:45.880071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:45.880241 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 02:21:45.880357 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 02:21:45.880530 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260808 02:21:45.881382 24811 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42689
I20260808 02:21:45.881485 24811 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:45.881753 24811 heartbeater.cc:511] Master 127.28.177.190:42689 requested a full tablet report, sending...
I20260808 02:21:45.882401 24533 ts_manager.cc:194] Registered new tserver with Master: 0ba88f3abfdd498d9b33d6010ff41364 (127.28.177.129:39465)
WARNING: no policy specified for kudu/127.28.177.130@KRBTEST.COM; defaulting to no policy
I20260808 02:21:45.883081 24533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.129@KRBTEST.COM'} at 127.28.177.129:41209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
Entry for principal kudu/127.28.177.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab.
I20260808 02:21:45.903851 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42689
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:45.989738 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:45.989959 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:45.989984 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:45.991117 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:45.991165 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:45.991192 24822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:45.992678 24822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.130.keytab
--principal=kudu/127.28.177.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:45.992975 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:45.993198 24822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:45.998739 24831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:45.998739 24832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:45.998914 24822 server_base.cc:1034] running on GCE node
W20260808 02:21:45.998745 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:45.999264 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:45.999575 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:46.000775 24822 hybrid_clock.cc:648] HybridClock initialized: now 1786155706000762 us; error 30 us; skew 500 ppm
Aug 08 02:21:45 dist-test-slave-3h5h krb5kdc[24476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:46.003803 24822 init.cc:377] Logged in from keytab as kudu/127.28.177.130@KRBTEST.COM (short username kudu)
I20260808 02:21:46.004267 24822 webserver.cc:460] Webserver started at http://127.28.177.130:45151/ using document root <none> and password file <none>
I20260808 02:21:46.004545 24822 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:46.004591 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:46.004767 24822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:46.005995 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "476fe192c90c499b9360dfb9c4aa5a1a"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "4c147f907e9c4ec9c095b79c3496dc43"
server_key_iv: "6d29230cf4e3ea9180380896e2d701ac"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.006383 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "476fe192c90c499b9360dfb9c4aa5a1a"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "4c147f907e9c4ec9c095b79c3496dc43"
server_key_iv: "6d29230cf4e3ea9180380896e2d701ac"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.007819 24822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:21:46.008710 24847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.008940 24822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:46.009043 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "476fe192c90c499b9360dfb9c4aa5a1a"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "4c147f907e9c4ec9c095b79c3496dc43"
server_key_iv: "6d29230cf4e3ea9180380896e2d701ac"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.009123 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:46.018100 24822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:46.020912 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:46.021070 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:46.021310 24822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:46.021647 24822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:46.021682 24822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.021733 24822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:46.021762 24822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.027676 24822 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:44873
I20260808 02:21:46.027763 25009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:44873 every 8 connection(s)
I20260808 02:21:46.028132 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:21:46.030118 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 24822
I20260808 02:21:46.030275 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:21:46.030504 29382 external_mini_cluster.cc:1428] Setting key 663e55ba54b664e3eabf9db61ebcf669
Aug 08 02:21:46 dist-test-slave-3h5h krb5kdc[24476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.130@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:46.039841 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.029104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37635 (local address 127.28.177.190:42689)
0808 02:21:46.029213 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.029215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.029431 (+ 216us) server_negotiation.cc:407] Connection header received
0808 02:21:46.029970 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.029971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.029986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.030060 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.031394 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.032138 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.033455 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.033653 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.033723 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.036368 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.036370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.036370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:46.036378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.037900 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:46.038662 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:46.038663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:46.038664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:46.038732 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:46.039134 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:46.039135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:46.039136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:46.039269 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.039352 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.039651 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.039738 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:21:46.040081 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.029077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42689 (local address 127.28.177.130:37635)
0808 02:21:46.029299 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.029337 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.029864 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.030114 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.030118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.030965 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:46.031284 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.031292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.032261 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.032262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.033363 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.033367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.033758 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.033760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.033913 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.034467 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:46.034477 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:46.036248 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:46.038006 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:46.038015 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:46.038018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:46.038537 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:46.038813 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:46.038814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:46.038814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:46.038935 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:46.039341 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:46.039349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:46.039506 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.039629 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.039820 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for kudu/127.28.177.131@KRBTEST.COM; defaulting to no policy
I20260808 02:21:46.040716 25010 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42689
I20260808 02:21:46.040812 25010 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:46.041043 25010 heartbeater.cc:511] Master 127.28.177.190:42689 requested a full tablet report, sending...
I20260808 02:21:46.041463 24533 ts_manager.cc:194] Registered new tserver with Master: 476fe192c90c499b9360dfb9c4aa5a1a (127.28.177.130:44873)
I20260808 02:21:46.041846 24533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.130@KRBTEST.COM'} at 127.28.177.130:37635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
Entry for principal kudu/127.28.177.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab.
I20260808 02:21:46.061169 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42689
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:46.150494 25019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:46.150719 25019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:46.150748 25019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:46.151957 25019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:46.152009 25019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:46.152036 25019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:46.153738 25019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.131.keytab
--principal=kudu/127.28.177.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:46.154012 25019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:46.154295 25019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:46.160038 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.160054 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.160164 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:46.160145 25019 server_base.cc:1034] running on GCE node
I20260808 02:21:46.160549 25019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:46.160907 25019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:46.162082 25019 hybrid_clock.cc:648] HybridClock initialized: now 1786155706162055 us; error 44 us; skew 500 ppm
Aug 08 02:21:46 dist-test-slave-3h5h krb5kdc[24476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:46.165016 25019 init.cc:377] Logged in from keytab as kudu/127.28.177.131@KRBTEST.COM (short username kudu)
I20260808 02:21:46.165540 25019 webserver.cc:460] Webserver started at http://127.28.177.131:40603/ using document root <none> and password file <none>
I20260808 02:21:46.165786 25019 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:46.165855 25019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:46.166023 25019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:46.167124 25019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "8c410c053b244313aaeb8a71ab4358c4"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "9f4b89f044071e7ca2715d8225dac11c"
server_key_iv: "4ee8def3baf19fea5fc2aacfaca38db3"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.167510 25019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "8c410c053b244313aaeb8a71ab4358c4"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "9f4b89f044071e7ca2715d8225dac11c"
server_key_iv: "4ee8def3baf19fea5fc2aacfaca38db3"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.169065 25019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:21:46.169870 25039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.170056 25019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:46.170171 25019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "8c410c053b244313aaeb8a71ab4358c4"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "9f4b89f044071e7ca2715d8225dac11c"
server_key_iv: "4ee8def3baf19fea5fc2aacfaca38db3"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.170248 25019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:46.186305 25019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:46.188629 25019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:46.188933 25019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:46.189220 25019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:46.189574 25019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:46.189635 25019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.189704 25019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:46.189735 25019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.196390 25019 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:45589
I20260808 02:21:46.196435 25210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:45589 every 8 connection(s)
I20260808 02:21:46.196880 25019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:46.197883 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 25019
I20260808 02:21:46.198014 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:21:46.198292 29382 external_mini_cluster.cc:1428] Setting key b561a3da6e2d3456885b77a80ff0eb36
Aug 08 02:21:46 dist-test-slave-3h5h krb5kdc[24476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.131@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:21:46.208441 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.197784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:48155 (local address 127.28.177.190:42689)
0808 02:21:46.197879 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.197882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.198172 (+ 290us) server_negotiation.cc:407] Connection header received
0808 02:21:46.199201 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.199203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.199223 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.199328 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.200445 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.201035 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.202074 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.202291 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.202372 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.204872 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.204874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.204875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:46.204888 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.206603 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:46.207151 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:46.207153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:46.207154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:46.207231 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:46.207560 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:46.207562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:46.207562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:46.207711 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.207809 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.208149 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.208265 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:21:46.208640 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.197684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:42689 (local address 127.28.177.131:48155)
0808 02:21:46.198007 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.198064 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.198963 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.199420 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.199425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.200115 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:46.200327 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.200333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.201135 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.201136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.201980 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.201982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.202382 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.202383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.202536 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.203069 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:21:46.203079 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:46.204700 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:46.206728 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:46.206739 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:46.206742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:46.207029 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:46.207321 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:21:46.207322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:21:46.207323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:21:46.207472 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:21:46.207797 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:46.207807 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:46.207983 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.208139 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.208337 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":122,"threads_started":1}
I20260808 02:21:46.209332 25212 heartbeater.cc:347] Connected to a master server at 127.28.177.190:42689
I20260808 02:21:46.209466 25212 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:46.209749 25212 heartbeater.cc:511] Master 127.28.177.190:42689 requested a full tablet report, sending...
I20260808 02:21:46.210264 24533 ts_manager.cc:194] Registered new tserver with Master: 8c410c053b244313aaeb8a71ab4358c4 (127.28.177.131:45589)
I20260808 02:21:46.210618 24533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.177.131@KRBTEST.COM'} at 127.28.177.131:48155
I20260808 02:21:46.221755 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:46.230320 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.223428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.28.177.190:42689)
0808 02:21:46.223509 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.223511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.223608 (+ 97us) server_negotiation.cc:407] Connection header received
0808 02:21:46.223651 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.223652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.223662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.223719 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.224391 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.224946 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.225694 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.225876 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.225929 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.227200 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.227202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.227202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:46.227210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.228796 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:46.229348 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:46.229350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:46.229350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:46.229414 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:46.229725 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:46.229727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:46.229727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:46.229943 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.230074 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.230157 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.230241 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:21:46.232256 24533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:46.234346 24533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:46.242080 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.237103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:44873 (local address 127.0.0.1:47040)
0808 02:21:46.237458 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.237506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.237576 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.238111 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.238112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.238114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.238203 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.238206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.239568 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.239569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.240608 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.240610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.241804 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.241805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.241875 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.241880 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.241938 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.241990 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":27,"thread_start_us":112,"threads_started":1}
I20260808 02:21:46.242192 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.237278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.28.177.130:44873)
0808 02:21:46.237932 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.237936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.237945 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:21:46.237984 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.237986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.238001 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.238107 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.238291 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.239438 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.240724 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.241699 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.241796 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.241857 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.241999 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.242051 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":102,"threads_started":1}
I20260808 02:21:46.242673 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.237427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.28.177.129:39465)
0808 02:21:46.237649 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.237665 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.237744 (+ 79us) server_negotiation.cc:407] Connection header received
0808 02:21:46.237919 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.237922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.237938 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.238013 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.238311 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.239013 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.240545 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.241463 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.241564 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.241631 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.242476 (+ 845us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.242558 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260808 02:21:46.242861 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.237129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:39465 (local address 127.0.0.1:51226)
0808 02:21:46.237698 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.237707 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.237813 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.238111 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.238112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.238116 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.238203 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.238213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.239123 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.239125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.240428 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.240431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.241682 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.241684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.241801 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.241821 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.241904 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.241979 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":8,"thread_start_us":99,"threads_started":1}
I20260808 02:21:46.242871 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.237104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:45589 (local address 127.0.0.1:40508)
0808 02:21:46.237483 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.237507 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.237591 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.238058 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.238062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.238064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.238142 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.238145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.240700 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.240701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.241463 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.241465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.242436 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.242438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.242500 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.242503 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.242533 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.242574 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":134,"threads_started":1}
I20260808 02:21:46.243328 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.237650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.28.177.131:45589)
0808 02:21:46.237848 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.237853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.237867 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:21:46.237913 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.237914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.237932 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.238235 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.238325 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.239444 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.241561 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.242314 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.242936 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.243153 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.243186 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.243240 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":128,"threads_started":1}
I20260808 02:21:46.243392 24914 tablet_service.cc:1467] Processing CreateTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 (DEFAULT_TABLE table=test-table [id=4f07793b00e84b7b8f1b09b2091cf86b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:46.243861 24914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d697d9fb78594e2ba4d1a34e19ef2760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.244166 25105 tablet_service.cc:1467] Processing CreateTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 (DEFAULT_TABLE table=test-table [id=4f07793b00e84b7b8f1b09b2091cf86b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:46.244529 25105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d697d9fb78594e2ba4d1a34e19ef2760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.245405 24717 tablet_service.cc:1467] Processing CreateTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 (DEFAULT_TABLE table=test-table [id=4f07793b00e84b7b8f1b09b2091cf86b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:46.245769 24717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d697d9fb78594e2ba4d1a34e19ef2760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.247694 25244 tablet_bootstrap.cc:492] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Bootstrap starting.
I20260808 02:21:46.248508 25244 tablet_bootstrap.cc:654] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.248756 25246 tablet_bootstrap.cc:492] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364: Bootstrap starting.
I20260808 02:21:46.248950 25245 tablet_bootstrap.cc:492] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Bootstrap starting.
I20260808 02:21:46.249267 25244 log.cc:824] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.249565 25246 tablet_bootstrap.cc:654] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.249759 25245 tablet_bootstrap.cc:654] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.250159 25246 log.cc:824] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.250255 25244 tablet_bootstrap.cc:492] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: No bootstrap required, opened a new log
I20260808 02:21:46.250340 25244 ts_tablet_manager.cc:1397] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:21:46.250391 25245 log.cc:824] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.251101 25246 tablet_bootstrap.cc:492] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364: No bootstrap required, opened a new log
I20260808 02:21:46.251175 25246 ts_tablet_manager.cc:1397] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:46.251207 25245 tablet_bootstrap.cc:492] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: No bootstrap required, opened a new log
I20260808 02:21:46.251276 25245 ts_tablet_manager.cc:1397] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:46.252514 25244 raft_consensus.cc:348] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.252714 25244 raft_consensus.cc:374] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.252751 25244 raft_consensus.cc:729] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476fe192c90c499b9360dfb9c4aa5a1a, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.252944 25244 consensus_queue.cc:260] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.253273 25244 ts_tablet_manager.cc:1428] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:46.253450 25010 heartbeater.cc:503] Master 127.28.177.190:42689 was elected leader, sending a full tablet report...
I20260808 02:21:46.253341 25246 raft_consensus.cc:348] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.253515 25246 raft_consensus.cc:374] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.253540 25246 raft_consensus.cc:729] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba88f3abfdd498d9b33d6010ff41364, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.253413 25245 raft_consensus.cc:348] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.253599 25245 raft_consensus.cc:374] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.253635 25245 raft_consensus.cc:729] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c410c053b244313aaeb8a71ab4358c4, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.253661 25246 consensus_queue.cc:260] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.254033 25246 ts_tablet_manager.cc:1428] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:46.254033 24811 heartbeater.cc:503] Master 127.28.177.190:42689 was elected leader, sending a full tablet report...
I20260808 02:21:46.253763 25245 consensus_queue.cc:260] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.254753 25245 ts_tablet_manager.cc:1428] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:46.254776 25212 heartbeater.cc:503] Master 127.28.177.190:42689 was elected leader, sending a full tablet report...
I20260808 02:21:46.257215 25250 raft_consensus.cc:482] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:46.257339 25250 raft_consensus.cc:504] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.257710 25250 leader_election.cc:288] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c410c053b244313aaeb8a71ab4358c4 (127.28.177.131:45589), 0ba88f3abfdd498d9b33d6010ff41364 (127.28.177.129:39465)
I20260808 02:21:46.260988 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.257883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:45589 (local address 127.28.177.130:33803)
0808 02:21:46.257949 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.257963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.258006 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.258133 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.258135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.258136 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.258229 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.258231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.258968 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.258969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.259752 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.259755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.260631 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.260632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.260785 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.260790 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.260869 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.260912 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:21:46.261026 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.257948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:33803 (local address 127.28.177.131:45589)
0808 02:21:46.257989 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.257991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.258001 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:46.258063 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.258064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.258071 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.258118 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.258320 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.258877 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.259867 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.260535 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.260614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.260623 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.260902 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.260958 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:21:46.261420 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.257971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:36177 (local address 127.28.177.129:39465)
0808 02:21:46.258016 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.258018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.258291 (+ 273us) server_negotiation.cc:407] Connection header received
0808 02:21:46.258324 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.258326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.258336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.258390 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.258516 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.259030 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.260465 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.261030 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.261102 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.261112 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.261329 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.261378 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:21:46.261427 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.257883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:39465 (local address 127.28.177.130:36177)
0808 02:21:46.258200 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.258210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.258265 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.258385 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.258385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.258387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.258455 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.258456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.259170 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.259171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.260356 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.260359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.261148 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.261149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.261232 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.261236 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.261315 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.261351 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":146,"thread_start_us":96,"threads_started":1}
I20260808 02:21:46.261475 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697d9fb78594e2ba4d1a34e19ef2760" candidate_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c410c053b244313aaeb8a71ab4358c4" is_pre_election: true
I20260808 02:21:46.261631 25138 raft_consensus.cc:2454] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476fe192c90c499b9360dfb9c4aa5a1a in term 0.
I20260808 02:21:46.261700 24751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697d9fb78594e2ba4d1a34e19ef2760" candidate_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba88f3abfdd498d9b33d6010ff41364" is_pre_election: true
I20260808 02:21:46.261833 24751 raft_consensus.cc:2454] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476fe192c90c499b9360dfb9c4aa5a1a in term 0.
I20260808 02:21:46.261886 24848 leader_election.cc:302] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476fe192c90c499b9360dfb9c4aa5a1a, 8c410c053b244313aaeb8a71ab4358c4; no voters:
I20260808 02:21:46.262024 25250 raft_consensus.cc:2781] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:46.262079 25250 raft_consensus.cc:482] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:46.262096 25250 raft_consensus.cc:3037] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:46.262810 25250 raft_consensus.cc:504] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.262959 25250 leader_election.cc:288] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [CANDIDATE]: Term 1 election: Requested vote from peers 8c410c053b244313aaeb8a71ab4358c4 (127.28.177.131:45589), 0ba88f3abfdd498d9b33d6010ff41364 (127.28.177.129:39465)
I20260808 02:21:46.263124 25138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697d9fb78594e2ba4d1a34e19ef2760" candidate_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c410c053b244313aaeb8a71ab4358c4"
I20260808 02:21:46.263124 24751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697d9fb78594e2ba4d1a34e19ef2760" candidate_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba88f3abfdd498d9b33d6010ff41364"
I20260808 02:21:46.263211 24751 raft_consensus.cc:3037] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:46.263211 25138 raft_consensus.cc:3037] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:46.264127 25138 raft_consensus.cc:2454] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476fe192c90c499b9360dfb9c4aa5a1a in term 1.
I20260808 02:21:46.264127 24751 raft_consensus.cc:2454] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476fe192c90c499b9360dfb9c4aa5a1a in term 1.
I20260808 02:21:46.264319 24854 leader_election.cc:302] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba88f3abfdd498d9b33d6010ff41364, 476fe192c90c499b9360dfb9c4aa5a1a; no voters:
I20260808 02:21:46.264422 25250 raft_consensus.cc:2781] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:46.264585 25250 raft_consensus.cc:686] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 1 LEADER]: Becoming Leader. State: Replica: 476fe192c90c499b9360dfb9c4aa5a1a, State: Running, Role: LEADER
I20260808 02:21:46.264693 25250 consensus_queue.cc:237] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } }
I20260808 02:21:46.265571 24532 catalog_manager.cc:5495] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 476fe192c90c499b9360dfb9c4aa5a1a (127.28.177.130). New cstate: current_term: 1 leader_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476fe192c90c499b9360dfb9c4aa5a1a" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 44873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260808 02:21:46.271998 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.269828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.28.177.130:44873)
0808 02:21:46.269872 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.269874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.269882 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:21:46.269943 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.269944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.269960 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.270018 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:46.270118 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.270568 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.271305 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.271482 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.271534 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.271673 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:46.271771 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:46.271827 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.271858 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.271875 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:21:46.274102 25138 raft_consensus.cc:1261] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 1 FOLLOWER]: Refusing update from remote peer 476fe192c90c499b9360dfb9c4aa5a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:46.274101 24751 raft_consensus.cc:1261] T d697d9fb78594e2ba4d1a34e19ef2760 P 0ba88f3abfdd498d9b33d6010ff41364 [term 1 FOLLOWER]: Refusing update from remote peer 476fe192c90c499b9360dfb9c4aa5a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:46.274514 25250 consensus_queue.cc:1035] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba88f3abfdd498d9b33d6010ff41364" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:46.274678 25254 consensus_queue.cc:1035] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c410c053b244313aaeb8a71ab4358c4" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 45589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:46.277501 25259 mvcc.cc:205] Tried to move back new op lower bound from 7316093772895494144 to 7316093772861018112. Current Snapshot: MvccSnapshot[applied={T|T < 7316093772895494144 or (T in {7316093772895494144})}]
I20260808 02:21:46.279332 24532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:46.279435 24532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:46.280256 24532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e458267352845b590ddc72da038fbe8: Sending DeleteTablet for 3 replicas of tablet d697d9fb78594e2ba4d1a34e19ef2760
I20260808 02:21:46.280558 25105 tablet_service.cc:1514] Processing DeleteTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:46 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:40508
I20260808 02:21:46.280652 24717 tablet_service.cc:1514] Processing DeleteTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:46 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:51226
I20260808 02:21:46.280853 25274 tablet_replica.cc:330] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: stopping tablet replica
I20260808 02:21:46.280884 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:46 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:51226
I20260808 02:21:46.280961 25274 raft_consensus.cc:2229] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:46.280987 24914 tablet_service.cc:1514] Processing DeleteTablet for tablet d697d9fb78594e2ba4d1a34e19ef2760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:46 UTC) from {username='kudu', principal='kudu/127.28.177.190@KRBTEST.COM'} at 127.0.0.1:47040
I20260808 02:21:46.281068 25274 raft_consensus.cc:2258] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:46.281123 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 24625
I20260808 02:21:46.290820 25277 tablet_replica.cc:330] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: stopping tablet replica
I20260808 02:21:46.291000 25277 raft_consensus.cc:2229] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:21:46.291132 25277 raft_consensus.cc:2258] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:46.292372 25277 ts_tablet_manager.cc:1905] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:46.295285 25274 ts_tablet_manager.cc:1905] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:46.296974 25277 ts_tablet_manager.cc:1918] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:46.297073 25277 log.cc:1196] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/d697d9fb78594e2ba4d1a34e19ef2760
I20260808 02:21:46.297335 25274 ts_tablet_manager.cc:1918] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:46.297421 25277 ts_tablet_manager.cc:1939] T d697d9fb78594e2ba4d1a34e19ef2760 P 476fe192c90c499b9360dfb9c4aa5a1a: Deleting consensus metadata
I20260808 02:21:46.297427 25274 log.cc:1196] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/d697d9fb78594e2ba4d1a34e19ef2760
I20260808 02:21:46.297783 25274 ts_tablet_manager.cc:1939] T d697d9fb78594e2ba4d1a34e19ef2760 P 8c410c053b244313aaeb8a71ab4358c4: Deleting consensus metadata
I20260808 02:21:46.298023 24516 catalog_manager.cc:4841] TS 476fe192c90c499b9360dfb9c4aa5a1a (127.28.177.130:44873): tablet d697d9fb78594e2ba4d1a34e19ef2760 (table test-table [id=4f07793b00e84b7b8f1b09b2091cf86b]) successfully deleted
I20260808 02:21:46.298240 24515 catalog_manager.cc:4841] TS 8c410c053b244313aaeb8a71ab4358c4 (127.28.177.131:45589): tablet d697d9fb78594e2ba4d1a34e19ef2760 (table test-table [id=4f07793b00e84b7b8f1b09b2091cf86b]) successfully deleted
W20260808 02:21:46.300201 24519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.177.129:39465 (error 108)
I20260808 02:21:46.300805 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.300656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:39465 (local address 127.0.0.1:51242)
0808 02:21:46.300721 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.300764 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:21:46.300945 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 24822
W20260808 02:21:46.308137 24519 catalog_manager.cc:4584] TS 0ba88f3abfdd498d9b33d6010ff41364 (127.28.177.129:39465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d697d9fb78594e2ba4d1a34e19ef2760: Network error: Client connection negotiation failed: client connection to 127.28.177.129:39465: connect: Connection refused (error 111)
I20260808 02:21:46.314945 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 25019
I20260808 02:21:46.329383 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 24492
2026-08-08T02:21:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T02:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:46Z Disabled control of system clock
I20260808 02:21:46.367219 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:45971
--webserver_interface=127.28.177.190
--webserver_port=0
--builtin_ntp_servers=127.28.177.148:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:46.452636 25283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:46.452894 25283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:46.452919 25283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:46.454382 25283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:46.454453 25283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:46.454479 25283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:46.454519 25283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:46.454551 25283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:46.456066 25283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:45971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:45971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:46.456290 25283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:46.456537 25283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:46.461833 25293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.461796 25291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.461804 25289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:46.462023 25283 server_base.cc:1034] running on GCE node
I20260808 02:21:46.462441 25283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:46.462780 25283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:46.463929 25283 hybrid_clock.cc:648] HybridClock initialized: now 1786155706463894 us; error 55 us; skew 500 ppm
I20260808 02:21:46.465281 25283 webserver.cc:460] Webserver started at http://127.28.177.190:34095/ using document root <none> and password file <none>
I20260808 02:21:46.465489 25283 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:46.465530 25283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:46.465649 25283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:46.466722 25283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "7bc8ddf652bf4adf99fef74c947eeb48"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "164fe771861064096372efecf843a790"
server_key_iv: "b7a6d75e3139ee1e2c56e89e60f84f8d"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.467039 25283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "7bc8ddf652bf4adf99fef74c947eeb48"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "164fe771861064096372efecf843a790"
server_key_iv: "b7a6d75e3139ee1e2c56e89e60f84f8d"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.468374 25283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:21:46.469345 25299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.469597 25283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:46.469691 25283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "7bc8ddf652bf4adf99fef74c947eeb48"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "164fe771861064096372efecf843a790"
server_key_iv: "b7a6d75e3139ee1e2c56e89e60f84f8d"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.469746 25283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:46.495433 25283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:46.495769 25283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:46.495883 25283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:46.500490 25283 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:45971
I20260808 02:21:46.500532 25381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:45971 every 8 connection(s)
I20260808 02:21:46.500952 25283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:46.502276 25382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.502969 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 25283
I20260808 02:21:46.503091 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:46.503338 29382 external_mini_cluster.cc:1428] Setting key 3c65cd5bac3a4e234958c5c6d2698dba
I20260808 02:21:46.506011 25382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48: Bootstrap starting.
I20260808 02:21:46.506810 25382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.507167 25382 log.cc:824] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.507951 25382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48: No bootstrap required, opened a new log
I20260808 02:21:46.510212 25382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45971 } }
I20260808 02:21:46.510438 25382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.510494 25382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc8ddf652bf4adf99fef74c947eeb48, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.510635 25382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45971 } }
I20260808 02:21:46.510736 25382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:46.510773 25382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:46.510823 25382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:46.511180 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.505135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.28.177.190:45971)
0808 02:21:46.505587 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.505596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.505633 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:21:46.506181 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.506191 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.506271 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.506448 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.506738 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.508078 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.508893 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.509530 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.509700 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.510358 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.510363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.510363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:46.510579 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.510624 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:46.510638 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.510721 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.510925 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.511007 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":106,"threads_started":1}
I20260808 02:21:46.511546 25382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45971 } }
I20260808 02:21:46.511654 25382 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc8ddf652bf4adf99fef74c947eeb48; no voters:
I20260808 02:21:46.511862 25382 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:46.511910 25391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:46.512072 25391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [term 1 LEADER]: Becoming Leader. State: Replica: 7bc8ddf652bf4adf99fef74c947eeb48, State: Running, Role: LEADER
I20260808 02:21:46.512184 25382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:46.512203 25391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45971 } }
I20260808 02:21:46.512544 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc8ddf652bf4adf99fef74c947eeb48. Latest consensus state: current_term: 1 leader_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45971 } } }
I20260808 02:21:46.512650 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:46.512562 25392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc8ddf652bf4adf99fef74c947eeb48" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 45971 } } }
I20260808 02:21:46.512813 25392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:46.512954 25400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:46.513520 25400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:46.515139 25400 catalog_manager.cc:1347] Generated new cluster ID: ba17a098850a4ab9b096ed90e387b286
I20260808 02:21:46.515215 25400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:46.525944 25400 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:46.526492 25400 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:46.531615 25400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48: Generated new TSK 0
I20260808 02:21:46.531839 25400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:46.533751 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.129:0
--local_ip_for_outbound_sockets=127.28.177.129
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45971
--builtin_ntp_servers=127.28.177.148:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:21:46.621935 25416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:46.622149 25416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:46.622174 25416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:46.623383 25416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:46.623427 25416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:46.623453 25416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.129
I20260808 02:21:46.624882 25416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.177.129
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.129
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:46.625151 25416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:46.625509 25416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:46.632747 25424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.632779 25429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.632747 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:46.633235 25416 server_base.cc:1034] running on GCE node
I20260808 02:21:46.633467 25416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:46.633788 25416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:46.634953 25416 hybrid_clock.cc:648] HybridClock initialized: now 1786155706634942 us; error 30 us; skew 500 ppm
I20260808 02:21:46.636317 25416 webserver.cc:460] Webserver started at http://127.28.177.129:35583/ using document root <none> and password file <none>
I20260808 02:21:46.636583 25416 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:46.636673 25416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:46.636868 25416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:46.638021 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data/instance:
uuid: "4708b73ecafe446f80ffd3702000f2e1"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "6ad3c16e1c4c55c3eca100fd7ce3b20b"
server_key_iv: "1cc2856160fc487a3881d514ff37732e"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.638406 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance:
uuid: "4708b73ecafe446f80ffd3702000f2e1"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "6ad3c16e1c4c55c3eca100fd7ce3b20b"
server_key_iv: "1cc2856160fc487a3881d514ff37732e"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.639802 25416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:21:46.640635 25437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.640897 25416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:46.641002 25416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal
uuid: "4708b73ecafe446f80ffd3702000f2e1"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "6ad3c16e1c4c55c3eca100fd7ce3b20b"
server_key_iv: "1cc2856160fc487a3881d514ff37732e"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.641078 25416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:46.679648 25416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:46.680117 25416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:46.680419 25416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:46.680755 25416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:46.681183 25416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:46.681222 25416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.681246 25416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:46.681294 25416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.687393 25416 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.129:40423
I20260808 02:21:46.687476 25594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.129:40423 every 8 connection(s)
I20260808 02:21:46.688069 25416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/data/info.pb
I20260808 02:21:46.690586 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 25416
I20260808 02:21:46.690732 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-0/wal/instance
I20260808 02:21:46.690951 29382 external_mini_cluster.cc:1428] Setting key 40f9eb4436667fe9c68b2ad756c99821
I20260808 02:21:46.692104 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.130:0
--local_ip_for_outbound_sockets=127.28.177.130
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45971
--builtin_ntp_servers=127.28.177.148:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:21:46.694880 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.689055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.129:43687 (local address 127.28.177.190:45971)
0808 02:21:46.689192 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.689195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.689500 (+ 305us) server_negotiation.cc:407] Connection header received
0808 02:21:46.689996 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.689998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.690013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.690081 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.691039 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.691682 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.693208 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.693400 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.693454 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.694537 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.694539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.694539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:46.694544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.694562 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:46.694566 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.694642 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.694764 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.694796 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:21:46.695161 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.689105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45971 (local address 127.28.177.129:43687)
0808 02:21:46.689359 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.689397 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.689902 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.690138 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.690141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.690149 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:46.690919 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.690927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.691797 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.691798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.693103 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.693107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.693486 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.693487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.693664 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.694380 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:46.694381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:46.694412 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:46.694438 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:46.694656 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:46.694662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:46.694665 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.694755 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.694871 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":135,"threads_started":1}
I20260808 02:21:46.695641 25595 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45971
I20260808 02:21:46.695739 25595 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:46.695952 25595 heartbeater.cc:511] Master 127.28.177.190:45971 requested a full tablet report, sending...
I20260808 02:21:46.696494 25325 ts_manager.cc:194] Registered new tserver with Master: 4708b73ecafe446f80ffd3702000f2e1 (127.28.177.129:40423)
I20260808 02:21:46.697232 25325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.129:43687
W20260808 02:21:46.779783 25599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:46.780038 25599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:46.780078 25599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:46.781296 25599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:46.781365 25599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:46.781406 25599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.130
I20260808 02:21:46.782828 25599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.177.130
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.130
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:46.783114 25599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:46.783444 25599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:46.788774 25608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.788767 25606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.788767 25610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:46.789119 25599 server_base.cc:1034] running on GCE node
I20260808 02:21:46.789388 25599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:46.789633 25599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:46.790902 25599 hybrid_clock.cc:648] HybridClock initialized: now 1786155706790874 us; error 27 us; skew 500 ppm
I20260808 02:21:46.792562 25599 webserver.cc:460] Webserver started at http://127.28.177.130:44711/ using document root <none> and password file <none>
I20260808 02:21:46.792958 25599 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:46.793052 25599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:46.793222 25599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:46.794502 25599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data/instance:
uuid: "42d95f6546744e3782c7f835da0a7b05"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "e331b60461b881423c10d0145f0c9574"
server_key_iv: "1abbfb9f5eb7d6d09f2a3a7359fd9deb"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.794955 25599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance:
uuid: "42d95f6546744e3782c7f835da0a7b05"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "e331b60461b881423c10d0145f0c9574"
server_key_iv: "1abbfb9f5eb7d6d09f2a3a7359fd9deb"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.796571 25599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:21:46.797473 25616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.797653 25599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 02:21:46.797750 25599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal
uuid: "42d95f6546744e3782c7f835da0a7b05"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "e331b60461b881423c10d0145f0c9574"
server_key_iv: "1abbfb9f5eb7d6d09f2a3a7359fd9deb"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.797842 25599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:46.809432 25599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:46.809823 25599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:46.809991 25599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:46.810245 25599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:46.810608 25599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:46.810667 25599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.810712 25599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:46.810760 25599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.817503 25599 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.130:37097
I20260808 02:21:46.817549 25775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.130:37097 every 8 connection(s)
I20260808 02:21:46.817912 25599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/data/info.pb
I20260808 02:21:46.818939 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 25599
I20260808 02:21:46.819070 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal/instance
I20260808 02:21:46.819303 29382 external_mini_cluster.cc:1428] Setting key c91b9c2e4b92ab68163afa3e7526bf5e
I20260808 02:21:46.820638 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.177.131:0
--local_ip_for_outbound_sockets=127.28.177.131
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45971
--builtin_ntp_servers=127.28.177.148:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:21:46.824760 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.819034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:37849 (local address 127.28.177.190:45971)
0808 02:21:46.819109 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.819112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.819347 (+ 235us) server_negotiation.cc:407] Connection header received
0808 02:21:46.820049 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.820051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.820066 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.820117 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.821138 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.821669 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.822999 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.823357 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.823452 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.824434 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.824436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.824436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:46.824442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.824460 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:46.824464 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.824529 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.824603 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.824634 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:21:46.824910 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.818886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45971 (local address 127.28.177.130:37849)
0808 02:21:46.819197 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.819245 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.819928 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.820193 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.820197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.820208 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:46.820996 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.821004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.821781 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.821782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.822895 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.822898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.823484 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.823485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.823620 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.824283 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:46.824283 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:46.824314 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:46.824340 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:46.824530 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:46.824532 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:46.824535 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.824582 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.824673 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260808 02:21:46.825469 25776 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45971
I20260808 02:21:46.825585 25776 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:46.825819 25776 heartbeater.cc:511] Master 127.28.177.190:45971 requested a full tablet report, sending...
I20260808 02:21:46.826220 25325 ts_manager.cc:194] Registered new tserver with Master: 42d95f6546744e3782c7f835da0a7b05 (127.28.177.130:37097)
I20260808 02:21:46.826761 25325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.130:37849
W20260808 02:21:46.908416 25780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:46.908638 25780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:46.908660 25780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:46.909799 25780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:46.909868 25780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:46.909894 25780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.177.131
I20260808 02:21:46.911355 25780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.177.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.177.131
--webserver_port=0
--tserver_master_addrs=127.28.177.190:45971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.177.131
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:46.911568 25780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:46.911826 25780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:46.917526 25788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.917526 25787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:46.917553 25792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:46.917790 25780 server_base.cc:1034] running on GCE node
I20260808 02:21:46.918004 25780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:46.918217 25780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:46.919368 25780 hybrid_clock.cc:648] HybridClock initialized: now 1786155706919355 us; error 31 us; skew 500 ppm
I20260808 02:21:46.920706 25780 webserver.cc:460] Webserver started at http://127.28.177.131:44453/ using document root <none> and password file <none>
I20260808 02:21:46.920985 25780 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:46.921033 25780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:46.921192 25780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:46.922319 25780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data/instance:
uuid: "6bf1a4f342104488aed5501d9032c2b9"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "a86e962c6f7d4db89dc765e9200bbf98"
server_key_iv: "91210a2178e720e3757de574f2fc80d0"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.922698 25780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance:
uuid: "6bf1a4f342104488aed5501d9032c2b9"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "a86e962c6f7d4db89dc765e9200bbf98"
server_key_iv: "91210a2178e720e3757de574f2fc80d0"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.924126 25780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:46.924940 25798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.925113 25780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:46.925223 25780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal
uuid: "6bf1a4f342104488aed5501d9032c2b9"
format_stamp: "Formatted at 2026-08-08 02:21:46 on dist-test-slave-3h5h"
server_key: "a86e962c6f7d4db89dc765e9200bbf98"
server_key_iv: "91210a2178e720e3757de574f2fc80d0"
server_key_version: "encryptionkey@0"
I20260808 02:21:46.925312 25780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:46.940135 25780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:46.940505 25780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:46.940686 25780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:46.940979 25780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:21:46.941354 25780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:21:46.941412 25780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.941455 25780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:21:46.941500 25780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:46.947651 25780 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.131:37827
I20260808 02:21:46.947700 25962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.131:37827 every 8 connection(s)
I20260808 02:21:46.948062 25780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/data/info.pb
I20260808 02:21:46.953454 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.948876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.131:51713 (local address 127.28.177.190:45971)
0808 02:21:46.948964 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.948968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.949248 (+ 280us) server_negotiation.cc:407] Connection header received
0808 02:21:46.949745 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.949747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.949758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.949816 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.950496 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.950991 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.952027 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.952230 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.952290 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.953227 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.953228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.953228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:46.953233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.953247 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:46.953251 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.953298 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.953365 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.953388 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:21:46.953649 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.948816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:45971 (local address 127.28.177.131:51713)
0808 02:21:46.949100 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.949138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.949656 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.949870 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.949873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.949883 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:21:46.950409 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.950414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.951090 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.951091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.951935 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.951937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.952318 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.952319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.952430 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.953082 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:21:46.953082 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:21:46.953113 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:21:46.953139 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:21:46.953300 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:21:46.953302 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:21:46.953304 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.953350 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.953433 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":120,"threads_started":1}
I20260808 02:21:46.954294 25963 heartbeater.cc:347] Connected to a master server at 127.28.177.190:45971
I20260808 02:21:46.954407 25963 heartbeater.cc:464] Registering TS with master...
I20260808 02:21:46.954600 25963 heartbeater.cc:511] Master 127.28.177.190:45971 requested a full tablet report, sending...
I20260808 02:21:46.955051 25325 ts_manager.cc:194] Registered new tserver with Master: 6bf1a4f342104488aed5501d9032c2b9 (127.28.177.131:37827)
I20260808 02:21:46.955449 25325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.177.131:51713
I20260808 02:21:46.957473 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 25780
I20260808 02:21:46.957571 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal/instance
I20260808 02:21:46.957783 29382 external_mini_cluster.cc:1428] Setting key 8244bc0645576792b7ed4fc30a2195b2
I20260808 02:21:46.958760 29382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:21:46.963336 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.960215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.28.177.190:45971)
0808 02:21:46.960309 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.960312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.960469 (+ 157us) server_negotiation.cc:407] Connection header received
0808 02:21:46.960492 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.960493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.960502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.960553 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:46.960681 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.961255 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.961966 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.962145 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.962247 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.962928 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:46.962929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:46.962930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:46.962934 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:46.962948 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:46.962952 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:46.962994 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.963167 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.963190 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:21:46.965201 25325 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:21:46.967173 25325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:21:46.974350 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.969874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.28.177.130:37097)
0808 02:21:46.970312 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.970315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.970325 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:46.970365 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.970367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.970381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.970654 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.970724 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.971466 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.972996 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.973875 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.973996 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.974114 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.974204 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.974247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":94,"threads_started":1}
I20260808 02:21:46.974339 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.969751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.130:37097 (local address 127.0.0.1:34688)
0808 02:21:46.969935 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.969982 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.970065 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.970484 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.970485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.970488 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.970546 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.970549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.971568 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.971569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.972900 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.972904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.973996 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.973998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.974087 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.974102 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.974189 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.974251 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260808 02:21:46.974598 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.969780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40423 (local address 127.0.0.1:49712)
0808 02:21:46.969945 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.969981 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.970050 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.970427 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.970431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.970435 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.970522 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.970525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.972025 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.972026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.973230 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.973233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.974189 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.974191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.974254 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.974259 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.974291 (+ 32us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.974335 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260808 02:21:46.974660 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.970084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.28.177.129:40423)
0808 02:21:46.970250 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.970256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.970268 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:21:46.970311 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.970314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.970331 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.970418 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.970632 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.971922 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.973314 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.974064 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.974390 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.974492 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.974522 (+ 30us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.974568 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260808 02:21:46.975298 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.970004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:37827 (local address 127.0.0.1:55384)
0808 02:21:46.970244 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:46.970255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:46.970305 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:46.970692 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:46.970693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:46.970695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:46.970731 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.970732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.972082 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.972084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.973470 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:46.973474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.975074 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.975075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:46.975139 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.975144 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:21:46.975190 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 02:21:46.975229 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260808 02:21:46.975266 25680 tablet_service.cc:1467] Processing CreateTablet for tablet 10e25eb581ad43449e2d1a8d449f5dfd (DEFAULT_TABLE table=test-table [id=408307ff4120462eba69d9d67b08cf5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:46.975387 25498 tablet_service.cc:1467] Processing CreateTablet for tablet 10e25eb581ad43449e2d1a8d449f5dfd (DEFAULT_TABLE table=test-table [id=408307ff4120462eba69d9d67b08cf5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:46.975613 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:46.970222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.28.177.131:37827)
0808 02:21:46.970459 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:46.970464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:46.970474 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:21:46.970516 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:46.970518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:46.970536 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:46.970620 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:46.970813 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.972016 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.973588 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:46.974324 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:46.974388 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:46.975461 (+ 1073us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:46.975489 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 02:21:46.975531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260808 02:21:46.975723 25680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e25eb581ad43449e2d1a8d449f5dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.975761 25498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e25eb581ad43449e2d1a8d449f5dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.977726 25865 tablet_service.cc:1467] Processing CreateTablet for tablet 10e25eb581ad43449e2d1a8d449f5dfd (DEFAULT_TABLE table=test-table [id=408307ff4120462eba69d9d67b08cf5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:21:46.978096 25865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e25eb581ad43449e2d1a8d449f5dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:46.978926 25979 tablet_bootstrap.cc:492] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Bootstrap starting.
I20260808 02:21:46.979331 25980 tablet_bootstrap.cc:492] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1: Bootstrap starting.
I20260808 02:21:46.979715 25979 tablet_bootstrap.cc:654] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.980005 25980 tablet_bootstrap.cc:654] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.980343 25979 log.cc:824] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.980343 25980 log.cc:824] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.981271 25980 tablet_bootstrap.cc:492] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1: No bootstrap required, opened a new log
I20260808 02:21:46.981356 25980 ts_tablet_manager.cc:1397] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:21:46.981611 25979 tablet_bootstrap.cc:492] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: No bootstrap required, opened a new log
I20260808 02:21:46.981711 25979 ts_tablet_manager.cc:1397] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:21:46.981935 25983 tablet_bootstrap.cc:492] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Bootstrap starting.
I20260808 02:21:46.982733 25983 tablet_bootstrap.cc:654] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:46.983124 25983 log.cc:824] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:46.983677 25980 raft_consensus.cc:348] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:46.983883 25980 raft_consensus.cc:374] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.983932 25980 raft_consensus.cc:729] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4708b73ecafe446f80ffd3702000f2e1, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.984073 25980 consensus_queue.cc:260] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:46.984136 25983 tablet_bootstrap.cc:492] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: No bootstrap required, opened a new log
I20260808 02:21:46.984205 25983 ts_tablet_manager.cc:1397] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:21:46.984372 25980 ts_tablet_manager.cc:1428] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:21:46.984519 25595 heartbeater.cc:503] Master 127.28.177.190:45971 was elected leader, sending a full tablet report...
I20260808 02:21:46.984880 25979 raft_consensus.cc:348] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:46.985112 25979 raft_consensus.cc:374] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.985169 25979 raft_consensus.cc:729] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d95f6546744e3782c7f835da0a7b05, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.985347 25979 consensus_queue.cc:260] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:46.985651 25979 ts_tablet_manager.cc:1428] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 02:21:46.985780 25776 heartbeater.cc:503] Master 127.28.177.190:45971 was elected leader, sending a full tablet report...
I20260808 02:21:46.986905 25983 raft_consensus.cc:348] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:46.987052 25983 raft_consensus.cc:374] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:46.987076 25983 raft_consensus.cc:729] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf1a4f342104488aed5501d9032c2b9, State: Initialized, Role: FOLLOWER
I20260808 02:21:46.987160 25983 consensus_queue.cc:260] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:46.987367 25983 ts_tablet_manager.cc:1428] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:21:46.987437 25963 heartbeater.cc:503] Master 127.28.177.190:45971 was elected leader, sending a full tablet report...
I20260808 02:21:47.034972 25987 raft_consensus.cc:482] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:21:47.035161 25987 raft_consensus.cc:504] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:47.035602 25987 leader_election.cc:288] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4708b73ecafe446f80ffd3702000f2e1 (127.28.177.129:40423), 6bf1a4f342104488aed5501d9032c2b9 (127.28.177.131:37827)
I20260808 02:21:47.038767 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:47.035830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40423 (local address 127.28.177.130:54541)
0808 02:21:47.035905 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:47.035919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:47.035965 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:47.036178 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:47.036179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:47.036182 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:47.036279 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:47.036281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.037076 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.037078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:47.037917 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:47.037920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.038479 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.038481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:47.038561 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:47.038564 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:21:47.038620 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:21:47.038675 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260808 02:21:47.038818 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:47.035915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:42603 (local address 127.28.177.131:37827)
0808 02:21:47.035964 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:47.035966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:47.036225 (+ 259us) server_negotiation.cc:407] Connection header received
0808 02:21:47.036262 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:47.036264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:47.036275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:47.036340 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:47.036498 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.037050 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.038059 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.038447 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.038523 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:47.038531 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:47.038699 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:21:47.038743 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:21:47.038789 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:47.035882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.177.130:54541 (local address 127.28.177.129:40423)
0808 02:21:47.035936 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:47.035938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:47.035955 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:21:47.036074 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:47.036075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:47.036089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:47.036160 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:21:47.036366 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.036994 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.038020 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.038390 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.038447 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:47.038454 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:47.038664 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 02:21:47.038709 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 02:21:47.039075 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:47.035836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.131:37827 (local address 127.28.177.130:42603)
0808 02:21:47.036105 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:47.036117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:21:47.036184 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:21:47.036358 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:21:47.036359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:21:47.036361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:21:47.036429 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:47.036430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.037167 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.037168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:47.037937 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:21:47.037939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.038537 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.038538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:21:47.038615 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:47.038619 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:21:47.038680 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:21:47.038717 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":62,"thread_start_us":116,"threads_started":1}
I20260808 02:21:47.039183 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e25eb581ad43449e2d1a8d449f5dfd" candidate_uuid: "42d95f6546744e3782c7f835da0a7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4708b73ecafe446f80ffd3702000f2e1" is_pre_election: true
I20260808 02:21:47.039358 25525 raft_consensus.cc:2454] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42d95f6546744e3782c7f835da0a7b05 in term 0.
I20260808 02:21:47.039470 25896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e25eb581ad43449e2d1a8d449f5dfd" candidate_uuid: "42d95f6546744e3782c7f835da0a7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf1a4f342104488aed5501d9032c2b9" is_pre_election: true
I20260808 02:21:47.039633 25896 raft_consensus.cc:2454] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42d95f6546744e3782c7f835da0a7b05 in term 0.
I20260808 02:21:47.039630 25622 leader_election.cc:302] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d95f6546744e3782c7f835da0a7b05, 4708b73ecafe446f80ffd3702000f2e1; no voters:
I20260808 02:21:47.039795 25987 raft_consensus.cc:2781] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:21:47.039865 25987 raft_consensus.cc:482] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:21:47.039897 25987 raft_consensus.cc:3037] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:47.040634 25987 raft_consensus.cc:504] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:47.040798 25987 leader_election.cc:288] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [CANDIDATE]: Term 1 election: Requested vote from peers 4708b73ecafe446f80ffd3702000f2e1 (127.28.177.129:40423), 6bf1a4f342104488aed5501d9032c2b9 (127.28.177.131:37827)
I20260808 02:21:47.040974 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e25eb581ad43449e2d1a8d449f5dfd" candidate_uuid: "42d95f6546744e3782c7f835da0a7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4708b73ecafe446f80ffd3702000f2e1"
I20260808 02:21:47.040995 25896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e25eb581ad43449e2d1a8d449f5dfd" candidate_uuid: "42d95f6546744e3782c7f835da0a7b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf1a4f342104488aed5501d9032c2b9"
I20260808 02:21:47.041056 25525 raft_consensus.cc:3037] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:47.041083 25896 raft_consensus.cc:3037] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:47.041852 25896 raft_consensus.cc:2454] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42d95f6546744e3782c7f835da0a7b05 in term 1.
I20260808 02:21:47.041899 25525 raft_consensus.cc:2454] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42d95f6546744e3782c7f835da0a7b05 in term 1.
I20260808 02:21:47.042066 25622 leader_election.cc:302] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d95f6546744e3782c7f835da0a7b05, 4708b73ecafe446f80ffd3702000f2e1; no voters:
I20260808 02:21:47.042163 25987 raft_consensus.cc:2781] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:47.042333 25987 raft_consensus.cc:686] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 1 LEADER]: Becoming Leader. State: Replica: 42d95f6546744e3782c7f835da0a7b05, State: Running, Role: LEADER
I20260808 02:21:47.042412 25987 consensus_queue.cc:237] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } }
I20260808 02:21:47.043546 25322 catalog_manager.cc:5495] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42d95f6546744e3782c7f835da0a7b05 (127.28.177.130). New cstate: current_term: 1 leader_uuid: "42d95f6546744e3782c7f835da0a7b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d95f6546744e3782c7f835da0a7b05" member_type: VOTER last_known_addr { host: "127.28.177.130" port: 37097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 } health_report { overall_health: UNKNOWN } } }
I20260808 02:21:47.049777 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:47.047526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.28.177.130:37097)
0808 02:21:47.047581 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:47.047583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:47.047592 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:21:47.047632 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:47.047633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:47.047649 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:47.047721 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:21:47.047834 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.048313 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.049089 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:47.049302 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:47.049358 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:47.049487 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:21:47.049581 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:21:47.049643 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:47.049687 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 02:21:47.049704 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:21:47.051676 25896 raft_consensus.cc:1261] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 1 FOLLOWER]: Refusing update from remote peer 42d95f6546744e3782c7f835da0a7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:47.051699 25525 raft_consensus.cc:1261] T 10e25eb581ad43449e2d1a8d449f5dfd P 4708b73ecafe446f80ffd3702000f2e1 [term 1 FOLLOWER]: Refusing update from remote peer 42d95f6546744e3782c7f835da0a7b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:21:47.052040 25987 consensus_queue.cc:1035] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf1a4f342104488aed5501d9032c2b9" member_type: VOTER last_known_addr { host: "127.28.177.131" port: 37827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:47.052210 25991 consensus_queue.cc:1035] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4708b73ecafe446f80ffd3702000f2e1" member_type: VOTER last_known_addr { host: "127.28.177.129" port: 40423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:21:47.054921 25995 mvcc.cc:205] Tried to move back new op lower bound from 7316093776080760832 to 7316093776046858240. Current Snapshot: MvccSnapshot[applied={T|T < 7316093776080760832}]
I20260808 02:21:47.055691 25999 mvcc.cc:205] Tried to move back new op lower bound from 7316093776080760832 to 7316093776046858240. Current Snapshot: MvccSnapshot[applied={T|T < 7316093776080760832}]
I20260808 02:21:47.058276 25324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:47.058408 25324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:21:47.059350 25324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc8ddf652bf4adf99fef74c947eeb48: Sending DeleteTablet for 3 replicas of tablet 10e25eb581ad43449e2d1a8d449f5dfd
I20260808 02:21:47.059686 25680 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e25eb581ad43449e2d1a8d449f5dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:47 UTC) from {username='slave'} at 127.0.0.1:34688
I20260808 02:21:47.059752 25865 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e25eb581ad43449e2d1a8d449f5dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:21:47 UTC) from {username='slave'} at 127.0.0.1:55384
I20260808 02:21:47.059929 26012 tablet_replica.cc:330] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: stopping tablet replica
I20260808 02:21:47.060024 26012 raft_consensus.cc:2229] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:21:47.060209 26012 raft_consensus.cc:2258] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:47.060226 26013 tablet_replica.cc:330] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: stopping tablet replica
I20260808 02:21:47.060354 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 25416
I20260808 02:21:47.064442 26013 raft_consensus.cc:2229] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:21:47.064597 26012 ts_tablet_manager.cc:1905] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:47.064633 26013 raft_consensus.cc:2258] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:21:47.065435 26013 ts_tablet_manager.cc:1905] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:21:47.066543 26012 ts_tablet_manager.cc:1918] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:47.066632 26012 log.cc:1196] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-1/wal/wals/10e25eb581ad43449e2d1a8d449f5dfd
I20260808 02:21:47.066920 26012 ts_tablet_manager.cc:1939] T 10e25eb581ad43449e2d1a8d449f5dfd P 42d95f6546744e3782c7f835da0a7b05: Deleting consensus metadata
I20260808 02:21:47.067377 26013 ts_tablet_manager.cc:1918] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:21:47.067433 25302 catalog_manager.cc:4841] TS 42d95f6546744e3782c7f835da0a7b05 (127.28.177.130:37097): tablet 10e25eb581ad43449e2d1a8d449f5dfd (table test-table [id=408307ff4120462eba69d9d67b08cf5c]) successfully deleted
I20260808 02:21:47.067482 26013 log.cc:1196] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Deleting WAL directory at /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155444117209-29382-0/minicluster-data/ts-2/wal/wals/10e25eb581ad43449e2d1a8d449f5dfd
I20260808 02:21:47.067755 26013 ts_tablet_manager.cc:1939] T 10e25eb581ad43449e2d1a8d449f5dfd P 6bf1a4f342104488aed5501d9032c2b9: Deleting consensus metadata
I20260808 02:21:47.068265 25302 catalog_manager.cc:4841] TS 6bf1a4f342104488aed5501d9032c2b9 (127.28.177.131:37827): tablet 10e25eb581ad43449e2d1a8d449f5dfd (table test-table [id=408307ff4120462eba69d9d67b08cf5c]) successfully deleted
W20260808 02:21:47.073294 25304 connection.cc:535] client connection to 127.28.177.129:40423 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 02:21:47.073442 25304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:21:47.073626 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 25599
I20260808 02:21:47.078405 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:47.073805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.129:40423 (local address 127.0.0.1:49718)
0808 02:21:47.078280 (+ 4475us) negotiation.cc:107] Waiting for socket to connect
0808 02:21:47.078354 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4439}
W20260808 02:21:47.080225 25304 catalog_manager.cc:4584] TS 4708b73ecafe446f80ffd3702000f2e1 (127.28.177.129:40423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10e25eb581ad43449e2d1a8d449f5dfd: Network error: Client connection negotiation failed: client connection to 127.28.177.129:40423: connect: Connection refused (error 111)
I20260808 02:21:47.091084 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 25780
I20260808 02:21:47.103808 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 25283
2026-08-08T02:21:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (773 ms)
[----------] 6 tests from EncryptionPolicyTest (9582 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:47 dist-test-slave-3h5h krb5kdc[26018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:47 dist-test-slave-3h5h krb5kdc[26018](info): set up 2 sockets
Aug 08 02:21:47 dist-test-slave-3h5h krb5kdc[26018](info): commencing operation
krb5kdc: starting...
W20260808 02:21:48.347680 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:48 dist-test-slave-3h5h krb5kdc[26018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:48.489228 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:48.580658 26039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:48.580935 26039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:48.580962 26039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:48.582244 26039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:48.582295 26039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:48.582310 26039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:48.582324 26039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:48.582336 26039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:48.584049 26039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:48.584364 26039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:48.584628 26039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:48.590083 26047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:48.590085 26049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:48.590109 26046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:48.590727 26039 server_base.cc:1034] running on GCE node
I20260808 02:21:48.590984 26039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:48.591338 26039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:48.592518 26039 hybrid_clock.cc:648] HybridClock initialized: now 1786155708592503 us; error 68 us; skew 500 ppm
Aug 08 02:21:48 dist-test-slave-3h5h krb5kdc[26018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:48.600512 26039 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:48.601051 26039 webserver.cc:460] Webserver started at http://127.0.0.1:39575/ using document root <none> and password file <none>
I20260808 02:21:48.601265 26039 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:48.601310 26039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:48.601405 26039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:48.602491 26039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "270608263fb74d639060373301f7842f"
format_stamp: "Formatted at 2026-08-08 02:21:48 on dist-test-slave-3h5h"
server_key: "64d7a60cb597a610fbd594331c5d39ce"
server_key_iv: "fd3733404b9a8b9a4df99abab67b5a68"
server_key_version: "encryptionkey@0"
I20260808 02:21:48.602802 26039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "270608263fb74d639060373301f7842f"
format_stamp: "Formatted at 2026-08-08 02:21:48 on dist-test-slave-3h5h"
server_key: "64d7a60cb597a610fbd594331c5d39ce"
server_key_iv: "fd3733404b9a8b9a4df99abab67b5a68"
server_key_version: "encryptionkey@0"
I20260808 02:21:48.604151 26039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 02:21:48.604990 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:48.605158 26039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:48.605254 26039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "270608263fb74d639060373301f7842f"
format_stamp: "Formatted at 2026-08-08 02:21:48 on dist-test-slave-3h5h"
server_key: "64d7a60cb597a610fbd594331c5d39ce"
server_key_iv: "fd3733404b9a8b9a4df99abab67b5a68"
server_key_version: "encryptionkey@0"
I20260808 02:21:48.605314 26039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:48.620676 26039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:48.622838 26039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:48.622958 26039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:48.628149 26039 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42995
I20260808 02:21:48.628226 26138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42995 every 8 connection(s)
I20260808 02:21:48.628630 26039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:48.629732 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:48.633019 26139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f: Bootstrap starting.
I20260808 02:21:48.633687 26139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:48.634004 26139 log.cc:824] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:48.634845 26139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f: No bootstrap required, opened a new log
I20260808 02:21:48.635998 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 26039
I20260808 02:21:48.636097 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:48.636337 29382 external_mini_cluster.cc:1428] Setting key 4efd8c269fbd8c3ad1ffbe19367713e4
I20260808 02:21:48.636955 26139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270608263fb74d639060373301f7842f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260808 02:21:48.637162 26139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:48.637213 26139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270608263fb74d639060373301f7842f, State: Initialized, Role: FOLLOWER
I20260808 02:21:48.637353 26139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270608263fb74d639060373301f7842f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260808 02:21:48.637459 26139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:48.637504 26139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:48.637558 26139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:48.638514 26139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270608263fb74d639060373301f7842f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260808 02:21:48.638652 26139 leader_election.cc:302] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 270608263fb74d639060373301f7842f; no voters:
I20260808 02:21:48.638870 26139 leader_election.cc:288] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:48.638911 26148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:48.639021 26148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [term 1 LEADER]: Becoming Leader. State: Replica: 270608263fb74d639060373301f7842f, State: Running, Role: LEADER
I20260808 02:21:48.639109 26148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270608263fb74d639060373301f7842f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260808 02:21:48.639164 26139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:48.639477 26151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 270608263fb74d639060373301f7842f. Latest consensus state: current_term: 1 leader_uuid: "270608263fb74d639060373301f7842f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270608263fb74d639060373301f7842f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } } }
I20260808 02:21:48.639580 26151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:48.639793 26150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "270608263fb74d639060373301f7842f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270608263fb74d639060373301f7842f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } } }
I20260808 02:21:48.639842 26153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:48.640594 26153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:48.639864 26150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:48.643115 26153 catalog_manager.cc:1347] Generated new cluster ID: 2bd76e6160734ccc9125f3c3c6fd093b
I20260808 02:21:48.643213 26153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:21:48 dist-test-slave-3h5h krb5kdc[26018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:21:48.648216 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:48.637044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33533 (local address 127.0.0.1:42995)
0808 02:21:48.637269 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:48.637279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:48.637314 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:21:48.637435 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:48.637440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:48.637510 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:48.637657 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:48.638300 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:48.639569 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:48.640614 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:48.641217 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:48.641666 (+ 449us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:48.644212 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:48.644217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:48.644217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:48.644231 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:48.646390 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:48.646857 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:48.646861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:48.646862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:48.646990 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:48.647231 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:48.647232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:48.647233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:48.647590 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:48.647814 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:48.647897 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 02:21:48.648004 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260808 02:21:48.649458 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:48.672187 26153 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:48.673008 26153 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:48.673403 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:48.653224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34789 (local address 127.0.0.1:42995)
0808 02:21:48.653265 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:48.653267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:48.665156 (+ 11889us) server_negotiation.cc:407] Connection header received
0808 02:21:48.665207 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:48.665208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:48.665226 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:48.665308 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:48.666196 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:48.666991 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:48.668153 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:48.668488 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:48.668553 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:48.670285 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:48.670286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:48.670287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:48.670295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:48.671957 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:48.672475 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:48.672476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:48.672477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:48.672543 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:48.672801 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:48.672803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:48.672803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:48.672957 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:48.673161 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:48.673220 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 02:21:48.673292 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:21:48.684727 26153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 270608263fb74d639060373301f7842f: Generated new TSK 0
I20260808 02:21:48.685024 26153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:48.688767 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 26039
W20260808 02:21:48.703665 26016 connection.cc:535] client connection to 127.0.0.1:42995 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42995): Connection reset by peer (error 104)
2026-08-08T02:21:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:48 dist-test-slave-3h5h krb5kdc[26181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:48 dist-test-slave-3h5h krb5kdc[26181](info): set up 2 sockets
Aug 08 02:21:48 dist-test-slave-3h5h krb5kdc[26181](info): commencing operation
krb5kdc: starting...
W20260808 02:21:49.973526 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:50 dist-test-slave-3h5h krb5kdc[26181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:50.118284 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:50.206187 26198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:50.206473 26198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:50.206523 26198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:50.207723 26198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:50.207803 26198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:50.207827 26198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:50.207850 26198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:50.207878 26198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:50.209458 26198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:50.209757 26198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:50.210017 26198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:50.215425 26212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:50.215567 26198 server_base.cc:1034] running on GCE node
W20260808 02:21:50.215432 26209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:50.215432 26207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:50.216030 26198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:50.216349 26198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:50.217511 26198 hybrid_clock.cc:648] HybridClock initialized: now 1786155710217501 us; error 54 us; skew 500 ppm
Aug 08 02:21:50 dist-test-slave-3h5h krb5kdc[26181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:50.220359 26198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:50.220973 26198 webserver.cc:460] Webserver started at http://127.0.0.1:44123/ using document root <none> and password file <none>
I20260808 02:21:50.221242 26198 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:50.221314 26198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:50.221465 26198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:50.222625 26198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "d717549920334c249149d16a4850dc7e"
format_stamp: "Formatted at 2026-08-08 02:21:50 on dist-test-slave-3h5h"
server_key: "92df3e700644b57a075a7967e9e69b2a"
server_key_iv: "1bff68f327e69a43c582949d68e4eb02"
server_key_version: "encryptionkey@0"
I20260808 02:21:50.223014 26198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "d717549920334c249149d16a4850dc7e"
format_stamp: "Formatted at 2026-08-08 02:21:50 on dist-test-slave-3h5h"
server_key: "92df3e700644b57a075a7967e9e69b2a"
server_key_iv: "1bff68f327e69a43c582949d68e4eb02"
server_key_version: "encryptionkey@0"
I20260808 02:21:50.224507 26198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:50.225409 26231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:50.225592 26198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:21:50.225704 26198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "d717549920334c249149d16a4850dc7e"
format_stamp: "Formatted at 2026-08-08 02:21:50 on dist-test-slave-3h5h"
server_key: "92df3e700644b57a075a7967e9e69b2a"
server_key_iv: "1bff68f327e69a43c582949d68e4eb02"
server_key_version: "encryptionkey@0"
I20260808 02:21:50.225780 26198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:50.262898 26198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:50.265462 26198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:50.265697 26198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:50.270396 26198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39271
I20260808 02:21:50.270476 26302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39271 every 8 connection(s)
I20260808 02:21:50.270892 26198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:50.271886 26304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:50.274722 26304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e: Bootstrap starting.
I20260808 02:21:50.275156 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 26198
I20260808 02:21:50.275295 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:50.275504 29382 external_mini_cluster.cc:1428] Setting key b8f5145a2c6e9f502d70534dc3ccb100
I20260808 02:21:50.275506 26304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:50.275918 26304 log.cc:824] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:50.276875 26304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e: No bootstrap required, opened a new log
I20260808 02:21:50.279430 26304 raft_consensus.cc:348] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717549920334c249149d16a4850dc7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260808 02:21:50.279630 26304 raft_consensus.cc:374] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:50.279668 26304 raft_consensus.cc:729] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d717549920334c249149d16a4850dc7e, State: Initialized, Role: FOLLOWER
I20260808 02:21:50.279798 26304 consensus_queue.cc:260] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717549920334c249149d16a4850dc7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260808 02:21:50.279877 26304 raft_consensus.cc:388] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:50.279909 26304 raft_consensus.cc:482] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:50.279960 26304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:50.280670 26304 raft_consensus.cc:504] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717549920334c249149d16a4850dc7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260808 02:21:50.280800 26304 leader_election.cc:302] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d717549920334c249149d16a4850dc7e; no voters:
I20260808 02:21:50.281042 26304 leader_election.cc:288] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:50.281111 26313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:50.281288 26313 raft_consensus.cc:686] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [term 1 LEADER]: Becoming Leader. State: Replica: d717549920334c249149d16a4850dc7e, State: Running, Role: LEADER
I20260808 02:21:50.281415 26304 sys_catalog.cc:564] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:50.281404 26313 consensus_queue.cc:237] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717549920334c249149d16a4850dc7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260808 02:21:50.281857 26316 sys_catalog.cc:455] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d717549920334c249149d16a4850dc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717549920334c249149d16a4850dc7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } } }
I20260808 02:21:50.282032 26316 sys_catalog.cc:458] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:50.282197 26314 sys_catalog.cc:455] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d717549920334c249149d16a4850dc7e. Latest consensus state: current_term: 1 leader_uuid: "d717549920334c249149d16a4850dc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717549920334c249149d16a4850dc7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } } }
I20260808 02:21:50.282716 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:50.276412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42349 (local address 127.0.0.1:39271)
0808 02:21:50.276581 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:50.276591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:50.276632 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:21:50.277232 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:50.277240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:50.277294 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:50.277424 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:50.277732 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:50.278785 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:50.280090 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:50.280528 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:50.280628 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:50.281730 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:50.281735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:50.281735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:50.281981 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:50.282023 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:50.282041 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:50.282152 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:50.282425 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 02:21:50.282459 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260808 02:21:50.282823 26314 sys_catalog.cc:458] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e [sys.catalog]: This master's current role is: LEADER
W20260808 02:21:50.283847 26333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:21:50.283903 26333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:21:50.283955 26334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:50.283967 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:50.284142 26334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:50.286171 26334 catalog_manager.cc:1347] Generated new cluster ID: d4d5baa761a94e1aa39341fe04effe5a
I20260808 02:21:50.286233 26334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:50.288921 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:50.285601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57409 (local address 127.0.0.1:39271)
0808 02:21:50.285701 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:50.285703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:50.285721 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:21:50.285744 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:50.285745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:50.285756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:50.285821 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:50.286019 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:50.286654 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:50.287443 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:50.287696 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:50.287787 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:50.288575 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:50.288577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:50.288577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:50.288583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:50.288599 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:50.288604 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:50.288670 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:50.288781 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:21:50.288792 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:21:50.316439 26334 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:50.317348 26334 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:50.326838 26334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d717549920334c249149d16a4850dc7e: Generated new TSK 0
I20260808 02:21:50.327087 26334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:50.330819 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 26198
W20260808 02:21:50.339746 26178 connection.cc:535] client connection to 127.0.0.1:39271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39271): Connection reset by peer (error 104)
2026-08-08T02:21:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:50 dist-test-slave-3h5h krb5kdc[26347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:50 dist-test-slave-3h5h krb5kdc[26347](info): set up 2 sockets
Aug 08 02:21:50 dist-test-slave-3h5h krb5kdc[26347](info): commencing operation
krb5kdc: starting...
W20260808 02:21:51.565732 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:51 dist-test-slave-3h5h krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:51.701030 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:51.789927 26364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:51.790169 26364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:51.790192 26364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:51.791303 26364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:51.791339 26364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:51.791353 26364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:51.791366 26364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:51.791378 26364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:51.792820 26364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:51.793197 26364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:51.793401 26364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:51.798686 26375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:51.798681 26371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:51.798728 26373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:51.799100 26364 server_base.cc:1034] running on GCE node
I20260808 02:21:51.799363 26364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:51.799718 26364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:51.800925 26364 hybrid_clock.cc:648] HybridClock initialized: now 1786155711800895 us; error 46 us; skew 500 ppm
Aug 08 02:21:51 dist-test-slave-3h5h krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:51.804256 26364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:51.804788 26364 webserver.cc:460] Webserver started at http://127.0.0.1:37621/ using document root <none> and password file <none>
I20260808 02:21:51.805083 26364 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:51.805162 26364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:51.805347 26364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:51.806496 26364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "006ba87e01a74c3aa24c87ea46e6c6a5"
format_stamp: "Formatted at 2026-08-08 02:21:51 on dist-test-slave-3h5h"
server_key: "54422b21f830caba5df58cacdc5c8e62"
server_key_iv: "053632260c85016f69ce31e6ebfbb0f9"
server_key_version: "encryptionkey@0"
I20260808 02:21:51.806880 26364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "006ba87e01a74c3aa24c87ea46e6c6a5"
format_stamp: "Formatted at 2026-08-08 02:21:51 on dist-test-slave-3h5h"
server_key: "54422b21f830caba5df58cacdc5c8e62"
server_key_iv: "053632260c85016f69ce31e6ebfbb0f9"
server_key_version: "encryptionkey@0"
I20260808 02:21:51.808310 26364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:51.809216 26384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:51.809463 26364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:51.809583 26364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "006ba87e01a74c3aa24c87ea46e6c6a5"
format_stamp: "Formatted at 2026-08-08 02:21:51 on dist-test-slave-3h5h"
server_key: "54422b21f830caba5df58cacdc5c8e62"
server_key_iv: "053632260c85016f69ce31e6ebfbb0f9"
server_key_version: "encryptionkey@0"
I20260808 02:21:51.809662 26364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:51.813788 26364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:51.816190 26364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:51.816358 26364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:51.820905 26364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37305
I20260808 02:21:51.820969 26464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37305 every 8 connection(s)
I20260808 02:21:51.821337 26364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:51.822468 26465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:51.824880 26465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5: Bootstrap starting.
I20260808 02:21:51.825515 26465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:51.825817 26465 log.cc:824] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:51.826737 26465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5: No bootstrap required, opened a new log
I20260808 02:21:51.827605 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 26364
I20260808 02:21:51.827732 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:51.827921 29382 external_mini_cluster.cc:1428] Setting key 7e68010bd21ae09077dfa686f676a448
I20260808 02:21:51.829180 26465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260808 02:21:51.829363 26465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:51.829409 26465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006ba87e01a74c3aa24c87ea46e6c6a5, State: Initialized, Role: FOLLOWER
I20260808 02:21:51.829638 26465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260808 02:21:51.829788 26465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:51.829852 26465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:51.829912 26465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:51.830816 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:51.828532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34909 (local address 127.0.0.1:37305)
0808 02:21:51.828806 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:51.828814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:51.828923 (+ 109us) server_negotiation.cc:407] Connection header received
0808 02:21:51.829158 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:51.829165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:51.829193 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:51.829311 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:51.830257 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:51.830263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:51.830263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:51.830269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:51.830326 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:51.830342 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:51.830411 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:51.830538 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 02:21:51.830582 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":158,"threads_started":1}
I20260808 02:21:51.831029 26465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260808 02:21:51.831141 26465 leader_election.cc:302] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006ba87e01a74c3aa24c87ea46e6c6a5; no voters:
I20260808 02:21:51.831312 26465 leader_election.cc:288] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:51.831528 26472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:51.831569 26465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:51.831722 26472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [term 1 LEADER]: Becoming Leader. State: Replica: 006ba87e01a74c3aa24c87ea46e6c6a5, State: Running, Role: LEADER
I20260808 02:21:51.831880 26472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260808 02:21:51.832257 26472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006ba87e01a74c3aa24c87ea46e6c6a5. Latest consensus state: current_term: 1 leader_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260808 02:21:51.832285 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:51.832314 26473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006ba87e01a74c3aa24c87ea46e6c6a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260808 02:21:51.832432 26472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:51.832481 26473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:51.832715 26484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:51.833407 26484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:51.835438 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:51.834065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35211 (local address 127.0.0.1:37305)
0808 02:21:51.834101 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:51.834103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:51.834322 (+ 219us) server_negotiation.cc:407] Connection header received
0808 02:21:51.834355 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:51.834356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:51.834363 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:51.834434 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:51.835195 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:51.835197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:51.835197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:51.835202 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:51.835221 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:51.835226 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:51.835284 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:51.835359 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:21:51.835373 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:21:51.835804 26484 catalog_manager.cc:1347] Generated new cluster ID: 2d1167e15c1a4f3384bef0e57df8c17c
I20260808 02:21:51.835847 26484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:51.844046 26484 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:51.844233 26484 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:51.849689 26484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006ba87e01a74c3aa24c87ea46e6c6a5: Generated new TSK 0
I20260808 02:21:51.849956 26484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:51.860301 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 26364
2026-08-08T02:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1539 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:51 dist-test-slave-3h5h krb5kdc[26509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:51 dist-test-slave-3h5h krb5kdc[26509](info): set up 2 sockets
Aug 08 02:21:51 dist-test-slave-3h5h krb5kdc[26509](info): commencing operation
krb5kdc: starting...
W20260808 02:21:53.180629 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:53 dist-test-slave-3h5h krb5kdc[26509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:53.329002 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:53.416229 26536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:53.416458 26536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:53.416486 26536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:53.417675 26536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:53.417716 26536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:53.417732 26536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:53.417744 26536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:53.417757 26536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:53.419274 26536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:53.419502 26536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:53.419692 26536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:53.425249 26544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:53.425271 26541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:53.425251 26542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:53.425544 26536 server_base.cc:1034] running on GCE node
I20260808 02:21:53.425853 26536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:53.426213 26536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:53.427373 26536 hybrid_clock.cc:648] HybridClock initialized: now 1786155713427344 us; error 46 us; skew 500 ppm
Aug 08 02:21:53 dist-test-slave-3h5h krb5kdc[26509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:53.430141 26536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:53.430701 26536 webserver.cc:460] Webserver started at http://127.0.0.1:35607/ using document root <none> and password file <none>
I20260808 02:21:53.430971 26536 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:53.431051 26536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:53.431209 26536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:53.432304 26536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "3af9a471b35343c19e99f05dceb922e4"
format_stamp: "Formatted at 2026-08-08 02:21:53 on dist-test-slave-3h5h"
server_key: "6853f970a3b42f5c92157cb892fd09aa"
server_key_iv: "142a63c074d03f073d0457ef29f9fc77"
server_key_version: "encryptionkey@0"
I20260808 02:21:53.432682 26536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "3af9a471b35343c19e99f05dceb922e4"
format_stamp: "Formatted at 2026-08-08 02:21:53 on dist-test-slave-3h5h"
server_key: "6853f970a3b42f5c92157cb892fd09aa"
server_key_iv: "142a63c074d03f073d0457ef29f9fc77"
server_key_version: "encryptionkey@0"
I20260808 02:21:53.434139 26536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:53.435005 26553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:53.435216 26536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:53.435333 26536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "3af9a471b35343c19e99f05dceb922e4"
format_stamp: "Formatted at 2026-08-08 02:21:53 on dist-test-slave-3h5h"
server_key: "6853f970a3b42f5c92157cb892fd09aa"
server_key_iv: "142a63c074d03f073d0457ef29f9fc77"
server_key_version: "encryptionkey@0"
I20260808 02:21:53.435411 26536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:53.442265 26536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:53.444622 26536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:53.444793 26536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:53.449421 26536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46663
I20260808 02:21:53.449553 26637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46663 every 8 connection(s)
I20260808 02:21:53.449898 26536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:53.450876 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:53.453280 26640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4: Bootstrap starting.
I20260808 02:21:53.453860 26640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:53.454116 26640 log.cc:824] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:53.454926 26640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4: No bootstrap required, opened a new log
I20260808 02:21:53.455966 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 26536
I20260808 02:21:53.456106 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:53.456334 29382 external_mini_cluster.cc:1428] Setting key 4279d35a899e0576b83f5692b8d72380
I20260808 02:21:53.456910 26640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9a471b35343c19e99f05dceb922e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260808 02:21:53.457120 26640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:53.457175 26640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af9a471b35343c19e99f05dceb922e4, State: Initialized, Role: FOLLOWER
I20260808 02:21:53.457341 26640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9a471b35343c19e99f05dceb922e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260808 02:21:53.457448 26640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:53.457494 26640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:53.457544 26640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:53.458783 26640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9a471b35343c19e99f05dceb922e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260808 02:21:53.458917 26640 leader_election.cc:302] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af9a471b35343c19e99f05dceb922e4; no voters:
I20260808 02:21:53.459106 26640 leader_election.cc:288] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:53.459169 26646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:53.459321 26646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [term 1 LEADER]: Becoming Leader. State: Replica: 3af9a471b35343c19e99f05dceb922e4, State: Running, Role: LEADER
I20260808 02:21:53.459403 26640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:53.459417 26646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9a471b35343c19e99f05dceb922e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260808 02:21:53.459933 26646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af9a471b35343c19e99f05dceb922e4. Latest consensus state: current_term: 1 leader_uuid: "3af9a471b35343c19e99f05dceb922e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9a471b35343c19e99f05dceb922e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } } }
I20260808 02:21:53.460018 26646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:53.459949 26647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af9a471b35343c19e99f05dceb922e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af9a471b35343c19e99f05dceb922e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } } }
I20260808 02:21:53.460134 26647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:53.460511 26653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:53.461109 26653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:53.462994 26653 catalog_manager.cc:1347] Generated new cluster ID: 98771d5b0aa549e0b5140efb4a08b6c9
I20260808 02:21:53.463060 26653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:21:53 dist-test-slave-3h5h krb5kdc[26509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:21:53.469492 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:53.457022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38293 (local address 127.0.0.1:46663)
0808 02:21:53.457283 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:53.457292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:53.457318 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:21:53.457420 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:53.457426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:53.457500 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:53.457688 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:53.458532 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:53.460269 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:53.461509 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:53.462188 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:53.462308 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:53.465163 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:53.465168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:53.465168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:53.465185 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:53.467502 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:53.467950 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:53.467954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:53.467955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:53.468122 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:53.468349 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:53.468350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:53.468351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:53.468733 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:53.468972 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:53.469070 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:21:53.469238 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
I20260808 02:21:53.470753 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:53.480322 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:53.472295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48807 (local address 127.0.0.1:46663)
0808 02:21:53.472396 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:53.472398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:53.472534 (+ 136us) server_negotiation.cc:407] Connection header received
0808 02:21:53.472612 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:53.472612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:53.472624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:53.472685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:53.473421 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:53.474186 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:53.475413 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:53.475717 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:53.475781 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:53.477374 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:53.477376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:53.477376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:53.477384 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:53.479059 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:53.479504 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:53.479504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:53.479505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:53.479564 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:53.479775 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:53.479776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:53.479777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:53.479896 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:53.479986 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:53.480135 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 02:21:53.480225 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:21:53.486047 26653 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:53.486616 26653 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:53.492556 26653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af9a471b35343c19e99f05dceb922e4: Generated new TSK 0
I20260808 02:21:53.492902 26653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:53.500597 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 26536
2026-08-08T02:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1636 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:53 dist-test-slave-3h5h krb5kdc[26686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:53 dist-test-slave-3h5h krb5kdc[26686](info): set up 2 sockets
Aug 08 02:21:53 dist-test-slave-3h5h krb5kdc[26686](info): commencing operation
krb5kdc: starting...
W20260808 02:21:54.843641 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:54 dist-test-slave-3h5h krb5kdc[26686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:54.985339 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:55.078944 26706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:55.079332 26706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:55.079358 26706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:55.080505 26706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:55.080546 26706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:55.080560 26706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:55.080574 26706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:55.080585 26706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:55.082084 26706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:55.082304 26706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:55.082491 26706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:55.088063 26716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:55.088249 26718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:55.088269 26706 server_base.cc:1034] running on GCE node
W20260808 02:21:55.088045 26712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:55.088732 26706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:55.089164 26706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:55.090358 26706 hybrid_clock.cc:648] HybridClock initialized: now 1786155715090333 us; error 43 us; skew 500 ppm
Aug 08 02:21:55 dist-test-slave-3h5h krb5kdc[26686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:55.098191 26706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:55.098786 26706 webserver.cc:460] Webserver started at http://127.0.0.1:40201/ using document root <none> and password file <none>
I20260808 02:21:55.099032 26706 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:55.099081 26706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:55.099191 26706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:55.100390 26706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "b048812c0366415c80012195940d10e3"
format_stamp: "Formatted at 2026-08-08 02:21:55 on dist-test-slave-3h5h"
server_key: "8f1eae8985615a1d4796e4376df4ef3f"
server_key_iv: "0b0e31d7e3adca8de385c607466b1367"
server_key_version: "encryptionkey@0"
I20260808 02:21:55.100747 26706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "b048812c0366415c80012195940d10e3"
format_stamp: "Formatted at 2026-08-08 02:21:55 on dist-test-slave-3h5h"
server_key: "8f1eae8985615a1d4796e4376df4ef3f"
server_key_iv: "0b0e31d7e3adca8de385c607466b1367"
server_key_version: "encryptionkey@0"
I20260808 02:21:55.102393 26706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 02:21:55.103335 26726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:55.103600 26706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:55.103713 26706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "b048812c0366415c80012195940d10e3"
format_stamp: "Formatted at 2026-08-08 02:21:55 on dist-test-slave-3h5h"
server_key: "8f1eae8985615a1d4796e4376df4ef3f"
server_key_iv: "0b0e31d7e3adca8de385c607466b1367"
server_key_version: "encryptionkey@0"
I20260808 02:21:55.103796 26706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:55.130108 26706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:55.132877 26706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:55.133042 26706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:55.137372 26706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46139
I20260808 02:21:55.137524 26802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46139 every 8 connection(s)
I20260808 02:21:55.137849 26706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:55.138933 26803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:55.141418 26803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3: Bootstrap starting.
I20260808 02:21:55.142066 26803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:55.142462 26803 log.cc:824] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:55.142544 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 26706
I20260808 02:21:55.142719 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:55.142911 29382 external_mini_cluster.cc:1428] Setting key a53484a3af4b70376dbcce1d47dec515
I20260808 02:21:55.143330 26803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3: No bootstrap required, opened a new log
I20260808 02:21:55.146382 26803 raft_consensus.cc:348] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048812c0366415c80012195940d10e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260808 02:21:55.146579 26803 raft_consensus.cc:374] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:55.146611 26803 raft_consensus.cc:729] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b048812c0366415c80012195940d10e3, State: Initialized, Role: FOLLOWER
I20260808 02:21:55.146729 26803 consensus_queue.cc:260] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048812c0366415c80012195940d10e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260808 02:21:55.146798 26803 raft_consensus.cc:388] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:55.146831 26803 raft_consensus.cc:482] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:55.146886 26803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:55.147508 26803 raft_consensus.cc:504] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048812c0366415c80012195940d10e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260808 02:21:55.147639 26803 leader_election.cc:302] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b048812c0366415c80012195940d10e3; no voters:
I20260808 02:21:55.147811 26803 leader_election.cc:288] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:55.147892 26811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:55.148064 26811 raft_consensus.cc:686] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [term 1 LEADER]: Becoming Leader. State: Replica: b048812c0366415c80012195940d10e3, State: Running, Role: LEADER
I20260808 02:21:55.148069 26803 sys_catalog.cc:564] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:55.148197 26811 consensus_queue.cc:237] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048812c0366415c80012195940d10e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260808 02:21:55.148546 26811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b048812c0366415c80012195940d10e3. Latest consensus state: current_term: 1 leader_uuid: "b048812c0366415c80012195940d10e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048812c0366415c80012195940d10e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } } }
I20260808 02:21:55.148636 26811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:55.148794 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b048812c0366415c80012195940d10e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b048812c0366415c80012195940d10e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } } }
I20260808 02:21:55.148902 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:55.149016 26822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:55.149657 26822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:55.150189 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:55.143668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45299 (local address 127.0.0.1:46139)
0808 02:21:55.143895 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:55.143905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:55.143944 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:21:55.144755 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:55.144765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:55.144834 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:55.145032 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:55.145351 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:55.146824 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:55.147648 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:55.148359 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:55.148484 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:55.149290 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:55.149295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:55.149296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:55.149516 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:55.149562 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:55.149579 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:55.149645 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:55.149877 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 02:21:55.149967 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":114,"threads_started":1}
I20260808 02:21:55.151422 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:55.151867 26822 catalog_manager.cc:1347] Generated new cluster ID: 4943207ed14841a198d10fb076effd79
I20260808 02:21:55.151917 26822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:55.156659 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:55.153072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48159 (local address 127.0.0.1:46139)
0808 02:21:55.153114 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:55.153116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:55.153218 (+ 102us) server_negotiation.cc:407] Connection header received
0808 02:21:55.153279 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:55.153280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:55.153291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:55.153345 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:55.153481 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:55.154223 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:55.155288 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:55.155580 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:55.155644 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:55.156395 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:55.156396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:55.156396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:55.156401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:55.156423 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:55.156427 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:55.156478 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:55.156550 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 02:21:55.156572 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:21:55.163056 26822 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:55.163796 26822 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:55.167888 26822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b048812c0366415c80012195940d10e3: Generated new TSK 0
I20260808 02:21:55.168095 26822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:55.173733 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 26706
2026-08-08T02:21:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1674 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:55 dist-test-slave-3h5h krb5kdc[26863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:55 dist-test-slave-3h5h krb5kdc[26863](info): set up 2 sockets
Aug 08 02:21:55 dist-test-slave-3h5h krb5kdc[26863](info): commencing operation
krb5kdc: starting...
W20260808 02:21:56.518010 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:56 dist-test-slave-3h5h krb5kdc[26863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:56.668233 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:56.757278 26879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:56.757500 26879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:56.757525 26879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:56.758731 26879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:56.758778 26879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:56.758792 26879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:56.758805 26879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:56.758816 26879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:56.760390 26879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:56.760659 26879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:56.760943 26879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:56.766433 26889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:56.766606 26892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:56.766434 26888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:56.766829 26879 server_base.cc:1034] running on GCE node
I20260808 02:21:56.767236 26879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:56.767637 26879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:56.768875 26879 hybrid_clock.cc:648] HybridClock initialized: now 1786155716768777 us; error 78 us; skew 500 ppm
Aug 08 02:21:56 dist-test-slave-3h5h krb5kdc[26863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:56.772068 26879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:56.772573 26879 webserver.cc:460] Webserver started at http://127.0.0.1:33305/ using document root <none> and password file <none>
I20260808 02:21:56.772816 26879 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:56.773039 26879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:56.773162 26879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:56.774516 26879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "e5f5a98a02594664b4e4384282ada9ce"
format_stamp: "Formatted at 2026-08-08 02:21:56 on dist-test-slave-3h5h"
server_key: "33e85b4d0814eab363ed6722130bdd74"
server_key_iv: "7e2bf9d29a944ffbf131942e7806c287"
server_key_version: "encryptionkey@0"
I20260808 02:21:56.774896 26879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "e5f5a98a02594664b4e4384282ada9ce"
format_stamp: "Formatted at 2026-08-08 02:21:56 on dist-test-slave-3h5h"
server_key: "33e85b4d0814eab363ed6722130bdd74"
server_key_iv: "7e2bf9d29a944ffbf131942e7806c287"
server_key_version: "encryptionkey@0"
I20260808 02:21:56.776531 26879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:21:56.777727 26903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:56.778118 26879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:21:56.778265 26879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "e5f5a98a02594664b4e4384282ada9ce"
format_stamp: "Formatted at 2026-08-08 02:21:56 on dist-test-slave-3h5h"
server_key: "33e85b4d0814eab363ed6722130bdd74"
server_key_iv: "7e2bf9d29a944ffbf131942e7806c287"
server_key_version: "encryptionkey@0"
I20260808 02:21:56.778363 26879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:56.782218 26879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:56.785002 26879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:56.785140 26879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:56.789194 26879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39985
I20260808 02:21:56.789258 26980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39985 every 8 connection(s)
I20260808 02:21:56.789592 26879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:56.790776 26981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:56.793350 26981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce: Bootstrap starting.
I20260808 02:21:56.794011 26981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:56.794353 26981 log.cc:824] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:56.794776 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 26879
I20260808 02:21:56.794927 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:56.795284 29382 external_mini_cluster.cc:1428] Setting key 19c27167223ec09949c74d083921f75e
I20260808 02:21:56.795377 26981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce: No bootstrap required, opened a new log
I20260808 02:21:56.797895 26981 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f5a98a02594664b4e4384282ada9ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260808 02:21:56.798189 26981 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:56.798242 26981 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f5a98a02594664b4e4384282ada9ce, State: Initialized, Role: FOLLOWER
I20260808 02:21:56.798405 26981 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f5a98a02594664b4e4384282ada9ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260808 02:21:56.798527 26981 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:56.798568 26981 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:56.798632 26981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:56.798892 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:56.795913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60799 (local address 127.0.0.1:39985)
0808 02:21:56.796133 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:56.796143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:56.796171 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:21:56.797314 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:56.797324 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:56.797356 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:56.797527 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:56.798343 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:56.798347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:56.798348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:56.798353 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:56.798415 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:56.798430 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:56.798509 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:56.798673 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 02:21:56.798723 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260808 02:21:56.799839 26981 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f5a98a02594664b4e4384282ada9ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260808 02:21:56.799963 26981 leader_election.cc:302] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f5a98a02594664b4e4384282ada9ce; no voters:
I20260808 02:21:56.800163 26981 leader_election.cc:288] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:56.800194 26988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:56.800338 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:56.800360 26988 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [term 1 LEADER]: Becoming Leader. State: Replica: e5f5a98a02594664b4e4384282ada9ce, State: Running, Role: LEADER
I20260808 02:21:56.800449 26988 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f5a98a02594664b4e4384282ada9ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260808 02:21:56.800496 26981 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:56.800889 26988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f5a98a02594664b4e4384282ada9ce. Latest consensus state: current_term: 1 leader_uuid: "e5f5a98a02594664b4e4384282ada9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f5a98a02594664b4e4384282ada9ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } } }
I20260808 02:21:56.800972 26988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:56.800953 26989 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f5a98a02594664b4e4384282ada9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f5a98a02594664b4e4384282ada9ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } } }
I20260808 02:21:56.801115 26989 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:56.801390 26995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:56.802271 26995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:56.804141 26995 catalog_manager.cc:1347] Generated new cluster ID: c22b7205e3474e63b4a646eb3c76000b
I20260808 02:21:56.804212 26995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:21:56.809422 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:56.803272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42507 (local address 127.0.0.1:39985)
0808 02:21:56.803320 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:56.803323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:56.803481 (+ 158us) server_negotiation.cc:407] Connection header received
0808 02:21:56.803517 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:56.803518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:56.803524 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:56.803593 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:56.809116 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:56.809118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:56.809119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:21:56.809127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:56.809154 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:21:56.809161 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:56.809246 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:56.809317 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 02:21:56.809335 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:21:56.841037 26995 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:56.841338 26995 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:56.847689 26995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f5a98a02594664b4e4384282ada9ce: Generated new TSK 0
I20260808 02:21:56.847944 26995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:56.857095 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 26879
2026-08-08T02:21:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1677 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:56 dist-test-slave-3h5h krb5kdc[27022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:56 dist-test-slave-3h5h krb5kdc[27022](info): set up 2 sockets
Aug 08 02:21:56 dist-test-slave-3h5h krb5kdc[27022](info): commencing operation
krb5kdc: starting...
W20260808 02:21:58.225062 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.342s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:58 dist-test-slave-3h5h krb5kdc[27022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:58.369560 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:21:58.456826 27042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:21:58.457079 27042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:21:58.457104 27042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:21:58.458254 27042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:21:58.458297 27042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:21:58.458312 27042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:21:58.458323 27042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:21:58.458336 27042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:21:58.459841 27042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:21:58.460199 27042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:21:58.460394 27042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:21:58.465991 27052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:58.465991 27048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:21:58.466023 27049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:21:58.466284 27042 server_base.cc:1034] running on GCE node
I20260808 02:21:58.466629 27042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:21:58.467077 27042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:21:58.468277 27042 hybrid_clock.cc:648] HybridClock initialized: now 1786155718468231 us; error 63 us; skew 500 ppm
Aug 08 02:21:58 dist-test-slave-3h5h krb5kdc[27022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:21:58.471539 27042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:21:58.472090 27042 webserver.cc:460] Webserver started at http://127.0.0.1:37905/ using document root <none> and password file <none>
I20260808 02:21:58.472361 27042 fs_manager.cc:359] Metadata directory not provided
I20260808 02:21:58.472450 27042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:21:58.472692 27042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:21:58.474059 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "84c5675088874e0ca3c16bd2fc10f034"
format_stamp: "Formatted at 2026-08-08 02:21:58 on dist-test-slave-3h5h"
server_key: "3e963386295d982aa98e076b4c53bfbb"
server_key_iv: "a2c8a6efa03c89b7f07e69afed374299"
server_key_version: "encryptionkey@0"
I20260808 02:21:58.474521 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "84c5675088874e0ca3c16bd2fc10f034"
format_stamp: "Formatted at 2026-08-08 02:21:58 on dist-test-slave-3h5h"
server_key: "3e963386295d982aa98e076b4c53bfbb"
server_key_iv: "a2c8a6efa03c89b7f07e69afed374299"
server_key_version: "encryptionkey@0"
I20260808 02:21:58.476141 27042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:21:58.477092 27061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:21:58.477381 27042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:21:58.477547 27042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "84c5675088874e0ca3c16bd2fc10f034"
format_stamp: "Formatted at 2026-08-08 02:21:58 on dist-test-slave-3h5h"
server_key: "3e963386295d982aa98e076b4c53bfbb"
server_key_iv: "a2c8a6efa03c89b7f07e69afed374299"
server_key_version: "encryptionkey@0"
I20260808 02:21:58.477644 27042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:21:58.500655 27042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:21:58.503537 27042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:21:58.503749 27042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:21:58.508500 27042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39945
I20260808 02:21:58.508548 27151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39945 every 8 connection(s)
I20260808 02:21:58.508948 27042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:21:58.509914 27152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:21:58.512367 27152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034: Bootstrap starting.
I20260808 02:21:58.513041 27152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034: Neither blocks nor log segments found. Creating new log.
I20260808 02:21:58.513303 27152 log.cc:824] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034: Log is configured to *not* fsync() on all Append() calls
I20260808 02:21:58.514063 27152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034: No bootstrap required, opened a new log
I20260808 02:21:58.515722 27152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c5675088874e0ca3c16bd2fc10f034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260808 02:21:58.515872 27152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:21:58.515895 27152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c5675088874e0ca3c16bd2fc10f034, State: Initialized, Role: FOLLOWER
I20260808 02:21:58.516049 27152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c5675088874e0ca3c16bd2fc10f034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260808 02:21:58.516139 27152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:21:58.516163 27152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:21:58.516194 27152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:21:58.516693 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 27042
I20260808 02:21:58.516750 27152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c5675088874e0ca3c16bd2fc10f034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260808 02:21:58.516889 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:21:58.516893 27152 leader_election.cc:302] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c5675088874e0ca3c16bd2fc10f034; no voters:
I20260808 02:21:58.517100 27152 leader_election.cc:288] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:21:58.517135 29382 external_mini_cluster.cc:1428] Setting key 14bc19ac0377b20083a42d4166799591
I20260808 02:21:58.517176 27155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:21:58.517379 27152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:21:58.517416 27155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [term 1 LEADER]: Becoming Leader. State: Replica: 84c5675088874e0ca3c16bd2fc10f034, State: Running, Role: LEADER
I20260808 02:21:58.517611 27155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c5675088874e0ca3c16bd2fc10f034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260808 02:21:58.518116 27156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c5675088874e0ca3c16bd2fc10f034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c5675088874e0ca3c16bd2fc10f034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260808 02:21:58.518354 27156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:58.518292 27157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c5675088874e0ca3c16bd2fc10f034. Latest consensus state: current_term: 1 leader_uuid: "84c5675088874e0ca3c16bd2fc10f034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c5675088874e0ca3c16bd2fc10f034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260808 02:21:58.518599 27157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034 [sys.catalog]: This master's current role is: LEADER
I20260808 02:21:58.518814 27171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:21:58.519451 27171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:21:58.521251 27171 catalog_manager.cc:1347] Generated new cluster ID: d2605c8d31c74da68c975761df56049a
I20260808 02:21:58.521313 27171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:21:58 dist-test-slave-3h5h krb5kdc[27022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:21:58.530366 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:58.518291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:43939 (local address 127.0.0.1:39945)
0808 02:21:58.518497 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:58.518508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:58.518541 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:21:58.518682 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:58.518691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:58.518761 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:58.519824 (+ 1063us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:58.519940 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:58.521398 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:58.522543 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:58.523192 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:58.523313 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:58.526237 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:58.526243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:58.526243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:58.526257 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:58.528393 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:58.528899 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:58.528903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:58.528903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:58.529033 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:58.529300 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:58.529300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:58.529301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:58.529667 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:58.529813 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:58.530005 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:21:58.530126 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260808 02:21:58.531685 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:21:58.539249 27171 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:21:58.540161 27171 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:21:58.541669 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:21:58.533400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:53211 (local address 127.0.0.1:39945)
0808 02:21:58.533463 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:21:58.533464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:21:58.533702 (+ 238us) server_negotiation.cc:407] Connection header received
0808 02:21:58.533759 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:21:58.533761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:21:58.533772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:21:58.533841 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:21:58.534459 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:58.535286 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:58.536333 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:21:58.536668 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:21:58.536746 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:21:58.538284 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:21:58.538286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:21:58.538286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:21:58.538294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:21:58.540144 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:58.540620 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:58.540621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:58.540622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:58.540706 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:21:58.540991 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:21:58.540992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:21:58.540992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:21:58.541115 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:21:58.541217 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:21:58.541494 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 02:21:58.541570 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:21:58.553649 27171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c5675088874e0ca3c16bd2fc10f034: Generated new TSK 0
I20260808 02:21:58.553902 27171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:21:58.564626 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 27042
W20260808 02:21:58.579075 27018 connection.cc:535] client connection to 127.0.0.1:39945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39945): Connection reset by peer (error 104)
2026-08-08T02:21:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1710 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:21:58 dist-test-slave-3h5h krb5kdc[27196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:21:58 dist-test-slave-3h5h krb5kdc[27196](info): set up 2 sockets
Aug 08 02:21:58 dist-test-slave-3h5h krb5kdc[27196](info): commencing operation
krb5kdc: starting...
W20260808 02:21:59.815833 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:21:59 dist-test-slave-3h5h krb5kdc[27196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:21:59.957228 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:00.047237 27220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:00.047526 27220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:00.047571 27220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:00.048813 27220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:00.048933 27220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:00.048959 27220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:00.048981 27220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:00.049010 27220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:00.050633 27220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:00.050957 27220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:00.051220 27220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:00.057615 27231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:00.057622 27227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:00.057648 27228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:00.057644 27220 server_base.cc:1034] running on GCE node
I20260808 02:22:00.058238 27220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:00.058549 27220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:00.059739 27220 hybrid_clock.cc:648] HybridClock initialized: now 1786155720059717 us; error 47 us; skew 500 ppm
Aug 08 02:22:00 dist-test-slave-3h5h krb5kdc[27196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:00.062673 27220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:22:00.063194 27220 webserver.cc:460] Webserver started at http://127.0.0.1:39731/ using document root <none> and password file <none>
I20260808 02:22:00.063441 27220 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:00.063514 27220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:00.063687 27220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:00.064939 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "64fd1c926cf147a4ad556b4fef0950dd"
format_stamp: "Formatted at 2026-08-08 02:22:00 on dist-test-slave-3h5h"
server_key: "2d4a9653207b5d795b7bb90dffdf0830"
server_key_iv: "b6e11f86bf430d1806695bc1fddb8e2e"
server_key_version: "encryptionkey@0"
I20260808 02:22:00.065342 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "64fd1c926cf147a4ad556b4fef0950dd"
format_stamp: "Formatted at 2026-08-08 02:22:00 on dist-test-slave-3h5h"
server_key: "2d4a9653207b5d795b7bb90dffdf0830"
server_key_iv: "b6e11f86bf430d1806695bc1fddb8e2e"
server_key_version: "encryptionkey@0"
I20260808 02:22:00.066829 27220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:22:00.067811 27243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:00.068112 27220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:00.068308 27220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "64fd1c926cf147a4ad556b4fef0950dd"
format_stamp: "Formatted at 2026-08-08 02:22:00 on dist-test-slave-3h5h"
server_key: "2d4a9653207b5d795b7bb90dffdf0830"
server_key_iv: "b6e11f86bf430d1806695bc1fddb8e2e"
server_key_version: "encryptionkey@0"
I20260808 02:22:00.068405 27220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:00.093781 27220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:00.100085 27220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:00.100289 27220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:00.105067 27220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40757
I20260808 02:22:00.105204 27315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40757 every 8 connection(s)
I20260808 02:22:00.105556 27220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:00.106604 27316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:00.109480 27316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd: Bootstrap starting.
I20260808 02:22:00.110198 27316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:00.110548 27316 log.cc:824] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:00.111481 27316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd: No bootstrap required, opened a new log
I20260808 02:22:00.113447 27316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd1c926cf147a4ad556b4fef0950dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260808 02:22:00.113607 27316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:00.113633 27316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1c926cf147a4ad556b4fef0950dd, State: Initialized, Role: FOLLOWER
I20260808 02:22:00.113816 27316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd1c926cf147a4ad556b4fef0950dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260808 02:22:00.113927 27316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:00.113979 27316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:00.114045 27316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:00.114681 27316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd1c926cf147a4ad556b4fef0950dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260808 02:22:00.114818 27316 leader_election.cc:302] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64fd1c926cf147a4ad556b4fef0950dd; no voters:
I20260808 02:22:00.114914 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 27220
I20260808 02:22:00.115048 27316 leader_election.cc:288] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:00.115067 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:00.115116 27322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:00.115309 27322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1c926cf147a4ad556b4fef0950dd, State: Running, Role: LEADER
I20260808 02:22:00.115317 29382 external_mini_cluster.cc:1428] Setting key 0760bc790a51775371519327d5f5221a
I20260808 02:22:00.115434 27316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:00.115456 27322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd1c926cf147a4ad556b4fef0950dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260808 02:22:00.115892 27322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64fd1c926cf147a4ad556b4fef0950dd. Latest consensus state: current_term: 1 leader_uuid: "64fd1c926cf147a4ad556b4fef0950dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd1c926cf147a4ad556b4fef0950dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } } }
I20260808 02:22:00.116123 27322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:00.116060 27324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64fd1c926cf147a4ad556b4fef0950dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd1c926cf147a4ad556b4fef0950dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } } }
I20260808 02:22:00.116370 27324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:00.116976 27335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:00.117619 27335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:00.119535 27335 catalog_manager.cc:1347] Generated new cluster ID: 0d1d1f11804842abbf8c205ce8406fbf
I20260808 02:22:00.119601 27335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:00.123068 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:00.116093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:57567 (local address 127.0.0.1:40757)
0808 02:22:00.116311 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:00.116322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:00.116745 (+ 423us) server_negotiation.cc:407] Connection header received
0808 02:22:00.117017 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:00.117025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:00.117109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:00.117256 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:00.117571 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:00.119221 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:00.120478 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:00.121228 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:00.121363 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:00.122217 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:00.122222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:00.122222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:00.122461 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:00.122521 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:00.122540 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:00.122641 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:00.122749 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 02:22:00.122787 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260808 02:22:00.124256 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:00.133445 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:00.129205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:56113 (local address 127.0.0.1:40757)
0808 02:22:00.129258 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:00.129260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:00.129447 (+ 187us) server_negotiation.cc:407] Connection header received
0808 02:22:00.129523 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:00.129524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:00.129539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:00.129600 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:00.129803 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:00.130653 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:00.131853 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:00.132217 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:00.132292 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:00.133149 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:00.133150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:00.133151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:00.133156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:00.133174 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:00.133179 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:00.133241 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:00.133332 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 02:22:00.133348 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:22:00.154650 27335 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:00.155288 27335 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:00.161576 27335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64fd1c926cf147a4ad556b4fef0950dd: Generated new TSK 0
I20260808 02:22:00.161844 27335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:00.167528 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 27220
W20260808 02:22:00.184870 27194 connection.cc:535] client connection to 127.0.0.1:40757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40757): Connection reset by peer (error 104)
2026-08-08T02:22:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:00 dist-test-slave-3h5h krb5kdc[27364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:00 dist-test-slave-3h5h krb5kdc[27364](info): set up 2 sockets
Aug 08 02:22:00 dist-test-slave-3h5h krb5kdc[27364](info): commencing operation
krb5kdc: starting...
W20260808 02:22:01.471199 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:01 dist-test-slave-3h5h krb5kdc[27364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:22:01.616598 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:01.702232 27379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:01.702512 27379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:01.702557 27379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:01.703791 27379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:01.703871 27379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:01.703896 27379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:01.703926 27379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:01.703958 27379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:01.705631 27379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:01.705925 27379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:01.706178 27379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:01.711668 27390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:01.711841 27379 server_base.cc:1034] running on GCE node
W20260808 02:22:01.711659 27393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:01.711659 27386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:01.712394 27379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:01.712725 27379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:01.713933 27379 hybrid_clock.cc:648] HybridClock initialized: now 1786155721713931 us; error 58 us; skew 500 ppm
Aug 08 02:22:01 dist-test-slave-3h5h krb5kdc[27364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:01.716987 27379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:22:01.717545 27379 webserver.cc:460] Webserver started at http://127.0.0.1:33943/ using document root <none> and password file <none>
I20260808 02:22:01.717799 27379 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:01.717871 27379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:01.718034 27379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:01.719163 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "c5e8be22514041d38bc46319dbddde75"
format_stamp: "Formatted at 2026-08-08 02:22:01 on dist-test-slave-3h5h"
server_key: "b82958eba3c0a3a568304c35f5a72ca0"
server_key_iv: "949d1bef8251d0d523b935d1e677ad30"
server_key_version: "encryptionkey@0"
I20260808 02:22:01.719547 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "c5e8be22514041d38bc46319dbddde75"
format_stamp: "Formatted at 2026-08-08 02:22:01 on dist-test-slave-3h5h"
server_key: "b82958eba3c0a3a568304c35f5a72ca0"
server_key_iv: "949d1bef8251d0d523b935d1e677ad30"
server_key_version: "encryptionkey@0"
I20260808 02:22:01.721170 27379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:01.722259 27402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:01.722527 27379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:01.722658 27379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "c5e8be22514041d38bc46319dbddde75"
format_stamp: "Formatted at 2026-08-08 02:22:01 on dist-test-slave-3h5h"
server_key: "b82958eba3c0a3a568304c35f5a72ca0"
server_key_iv: "949d1bef8251d0d523b935d1e677ad30"
server_key_version: "encryptionkey@0"
I20260808 02:22:01.722750 27379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:01.726807 27379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:01.729501 27379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:01.729679 27379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:01.734709 27379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43693
I20260808 02:22:01.734807 27476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43693 every 8 connection(s)
I20260808 02:22:01.735213 27379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:01.736354 27479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:01.738864 27479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75: Bootstrap starting.
I20260808 02:22:01.739521 27479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:01.739841 27479 log.cc:824] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:01.740625 27479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75: No bootstrap required, opened a new log
I20260808 02:22:01.742318 27479 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e8be22514041d38bc46319dbddde75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260808 02:22:01.742475 27479 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:01.742499 27479 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e8be22514041d38bc46319dbddde75, State: Initialized, Role: FOLLOWER
I20260808 02:22:01.742659 27479 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e8be22514041d38bc46319dbddde75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260808 02:22:01.742743 27479 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:01.742765 27479 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:01.742817 27479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:01.742959 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 27379
I20260808 02:22:01.743193 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:01.743418 29382 external_mini_cluster.cc:1428] Setting key 920372c189ea898f421a661fdf8d068a
I20260808 02:22:01.743520 27479 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e8be22514041d38bc46319dbddde75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260808 02:22:01.743661 27479 leader_election.cc:302] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5e8be22514041d38bc46319dbddde75; no voters:
I20260808 02:22:01.743857 27479 leader_election.cc:288] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:01.743907 27485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:01.744127 27485 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [term 1 LEADER]: Becoming Leader. State: Replica: c5e8be22514041d38bc46319dbddde75, State: Running, Role: LEADER
I20260808 02:22:01.744181 27479 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:01.744257 27485 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e8be22514041d38bc46319dbddde75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } }
I20260808 02:22:01.744623 27491 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5e8be22514041d38bc46319dbddde75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e8be22514041d38bc46319dbddde75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } } }
I20260808 02:22:01.744611 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5e8be22514041d38bc46319dbddde75. Latest consensus state: current_term: 1 leader_uuid: "c5e8be22514041d38bc46319dbddde75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e8be22514041d38bc46319dbddde75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43693 } } }
I20260808 02:22:01.744709 27491 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:01.744716 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:01.744983 27498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:01.745586 27498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:01.746637 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:01.744113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:35547 (local address 127.0.0.1:43693)
0808 02:22:01.744298 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:01.744308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:01.744582 (+ 274us) server_negotiation.cc:407] Connection header received
0808 02:22:01.744726 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:01.744735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:01.744775 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:01.744939 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:01.746124 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:01.746129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:01.746129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:01.746134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:01.746195 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:01.746215 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:01.746284 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:01.746399 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:22:01.746448 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260808 02:22:01.747538 27498 catalog_manager.cc:1347] Generated new cluster ID: 4a7a1aa578bf4182a2bbb980e0f5e67e
I20260808 02:22:01.747637 27498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:01.748041 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:01.750921 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:01.749808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:33419 (local address 127.0.0.1:43693)
0808 02:22:01.749850 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:01.749853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:01.749864 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:01.749913 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:01.749914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:01.749920 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:01.749970 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:01.750679 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:01.750681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:01.750681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:01.750686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:01.750700 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:01.750705 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:01.750764 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:01.750821 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 02:22:01.750838 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:01.757993 27498 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:01.758244 27498 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:01.768457 27498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5e8be22514041d38bc46319dbddde75: Generated new TSK 0
I20260808 02:22:01.768707 27498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:01.774770 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 27379
2026-08-08T02:22:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:01 dist-test-slave-3h5h krb5kdc[27523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:01 dist-test-slave-3h5h krb5kdc[27523](info): set up 2 sockets
Aug 08 02:22:01 dist-test-slave-3h5h krb5kdc[27523](info): commencing operation
krb5kdc: starting...
W20260808 02:22:03.071075 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:03 dist-test-slave-3h5h krb5kdc[27523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:22:03.215811 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:03.306372 27542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:03.306674 27542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:03.306716 27542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:03.307940 27542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:03.308022 27542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:03.308050 27542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:03.308081 27542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:03.308109 27542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:03.309955 27542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:03.310298 27542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:03.310552 27542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:03.316452 27551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:03.316455 27554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:03.316448 27549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:03.316762 27542 server_base.cc:1034] running on GCE node
I20260808 02:22:03.317140 27542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:03.317421 27542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:03.318608 27542 hybrid_clock.cc:648] HybridClock initialized: now 1786155723318590 us; error 50 us; skew 500 ppm
Aug 08 02:22:03 dist-test-slave-3h5h krb5kdc[27523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:03.321622 27542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:22:03.322155 27542 webserver.cc:460] Webserver started at http://127.0.0.1:39731/ using document root <none> and password file <none>
I20260808 02:22:03.322455 27542 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:03.322507 27542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:03.322703 27542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:03.323906 27542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "291432dfbd6845f1b433c6cbde87ebc8"
format_stamp: "Formatted at 2026-08-08 02:22:03 on dist-test-slave-3h5h"
server_key: "d28247bfb6d4b33ddbabab784caaf557"
server_key_iv: "8e121cb062a3a86f7303b6770986d76d"
server_key_version: "encryptionkey@0"
I20260808 02:22:03.324353 27542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "291432dfbd6845f1b433c6cbde87ebc8"
format_stamp: "Formatted at 2026-08-08 02:22:03 on dist-test-slave-3h5h"
server_key: "d28247bfb6d4b33ddbabab784caaf557"
server_key_iv: "8e121cb062a3a86f7303b6770986d76d"
server_key_version: "encryptionkey@0"
I20260808 02:22:03.326118 27542 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260808 02:22:03.326987 27569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:03.327282 27542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:03.327404 27542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "291432dfbd6845f1b433c6cbde87ebc8"
format_stamp: "Formatted at 2026-08-08 02:22:03 on dist-test-slave-3h5h"
server_key: "d28247bfb6d4b33ddbabab784caaf557"
server_key_iv: "8e121cb062a3a86f7303b6770986d76d"
server_key_version: "encryptionkey@0"
I20260808 02:22:03.327508 27542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:03.349453 27542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:03.352171 27542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:03.352381 27542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:03.357357 27542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38623
I20260808 02:22:03.357445 27642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38623 every 8 connection(s)
I20260808 02:22:03.357939 27542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:03.359035 27643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:03.361862 27643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8: Bootstrap starting.
I20260808 02:22:03.362501 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 27542
I20260808 02:22:03.362641 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:03.362715 27643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:03.362883 29382 external_mini_cluster.cc:1428] Setting key f8a86d959cfe9917f18181526680df7d
I20260808 02:22:03.363121 27643 log.cc:824] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:03.364086 27643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8: No bootstrap required, opened a new log
I20260808 02:22:03.366920 27643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291432dfbd6845f1b433c6cbde87ebc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260808 02:22:03.367138 27643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:03.367182 27643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291432dfbd6845f1b433c6cbde87ebc8, State: Initialized, Role: FOLLOWER
I20260808 02:22:03.367322 27643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291432dfbd6845f1b433c6cbde87ebc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260808 02:22:03.367405 27643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:03.367436 27643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:03.367484 27643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:03.368263 27643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291432dfbd6845f1b433c6cbde87ebc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260808 02:22:03.368398 27643 leader_election.cc:302] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 291432dfbd6845f1b433c6cbde87ebc8; no voters:
I20260808 02:22:03.368614 27643 leader_election.cc:288] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:03.368664 27652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:03.368855 27652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [term 1 LEADER]: Becoming Leader. State: Replica: 291432dfbd6845f1b433c6cbde87ebc8, State: Running, Role: LEADER
I20260808 02:22:03.369006 27643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:03.369024 27652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291432dfbd6845f1b433c6cbde87ebc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } }
I20260808 02:22:03.369527 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291432dfbd6845f1b433c6cbde87ebc8. Latest consensus state: current_term: 1 leader_uuid: "291432dfbd6845f1b433c6cbde87ebc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291432dfbd6845f1b433c6cbde87ebc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } } }
I20260808 02:22:03.369666 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:03.370163 27664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:03.370873 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "291432dfbd6845f1b433c6cbde87ebc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291432dfbd6845f1b433c6cbde87ebc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38623 } } }
I20260808 02:22:03.370965 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8 [sys.catalog]: This master's current role is: LEADER
Aug 08 02:22:03 dist-test-slave-3h5h krb5kdc[27523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:22:03.371515 27664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:03.373799 27664 catalog_manager.cc:1347] Generated new cluster ID: 88cd532a958242f2ae879113bf58fc92
I20260808 02:22:03.373857 27664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:03.376381 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:03.363509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:51025 (local address 127.0.0.1:38623)
0808 02:22:03.363759 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:03.363769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:03.363811 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:22:03.364390 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:03.364399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:03.364456 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:03.364583 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:03.365552 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:03.366658 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:03.367847 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:03.368290 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:03.368430 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:03.371957 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:03.371962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:03.371962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:03.371976 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:03.374337 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:03.374895 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:03.374899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:03.374899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:03.375010 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:03.375289 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:03.375290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:03.375290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:03.375593 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:03.375772 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:03.375937 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:22:03.376151 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260808 02:22:03.377408 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:03.386749 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:03.378973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:38665 (local address 127.0.0.1:38623)
0808 02:22:03.379014 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:03.379016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:03.379139 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:22:03.379227 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:03.379228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:03.379238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:03.379266 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:03.379954 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:03.380721 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:03.381755 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:03.382076 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:03.382116 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:03.383718 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:03.383720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:03.383720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:03.383728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:03.385478 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:03.385903 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:03.385904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:03.385905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:03.385959 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:03.386167 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:03.386168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:03.386168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:03.386289 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:03.386361 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:03.386574 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:22:03.386650 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:22:03.394879 27664 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:03.395659 27664 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:03.402863 27664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 291432dfbd6845f1b433c6cbde87ebc8: Generated new TSK 0
I20260808 02:22:03.403111 27664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:03.408725 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 27542
W20260808 02:22:03.418097 27521 connection.cc:535] client connection to 127.0.0.1:38623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38623): Connection reset by peer (error 104)
2026-08-08T02:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:03 dist-test-slave-3h5h krb5kdc[27691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:03 dist-test-slave-3h5h krb5kdc[27691](info): set up 2 sockets
Aug 08 02:22:03 dist-test-slave-3h5h krb5kdc[27691](info): commencing operation
krb5kdc: starting...
W20260808 02:22:04.650139 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:04 dist-test-slave-3h5h krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:22:04.792335 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:04.879145 27710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:04.879374 27710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:04.879397 27710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:04.880611 27710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:04.880656 27710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:04.880671 27710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:04.880684 27710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:04.880697 27710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:04.882241 27710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:04.882465 27710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:04.882664 27710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:04.888211 27720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:04.888130 27717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:04.888368 27715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:04.888445 27710 server_base.cc:1034] running on GCE node
I20260808 02:22:04.888811 27710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:04.889178 27710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:04.890394 27710 hybrid_clock.cc:648] HybridClock initialized: now 1786155724890377 us; error 30 us; skew 500 ppm
Aug 08 02:22:04 dist-test-slave-3h5h krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:04.893451 27710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:22:04.893985 27710 webserver.cc:460] Webserver started at http://127.0.0.1:39633/ using document root <none> and password file <none>
I20260808 02:22:04.894275 27710 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:04.894351 27710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:04.894495 27710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:04.895640 27710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "4b622094f2d24755adaf5c38490f5b6b"
format_stamp: "Formatted at 2026-08-08 02:22:04 on dist-test-slave-3h5h"
server_key: "6ab5e30f2b58b8eb475d787ccca039df"
server_key_iv: "8eee88704e7d3213c6acd66b7c37fc02"
server_key_version: "encryptionkey@0"
I20260808 02:22:04.896035 27710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "4b622094f2d24755adaf5c38490f5b6b"
format_stamp: "Formatted at 2026-08-08 02:22:04 on dist-test-slave-3h5h"
server_key: "6ab5e30f2b58b8eb475d787ccca039df"
server_key_iv: "8eee88704e7d3213c6acd66b7c37fc02"
server_key_version: "encryptionkey@0"
I20260808 02:22:04.897640 27710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:04.898487 27727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:04.898725 27710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:04.898844 27710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "4b622094f2d24755adaf5c38490f5b6b"
format_stamp: "Formatted at 2026-08-08 02:22:04 on dist-test-slave-3h5h"
server_key: "6ab5e30f2b58b8eb475d787ccca039df"
server_key_iv: "8eee88704e7d3213c6acd66b7c37fc02"
server_key_version: "encryptionkey@0"
I20260808 02:22:04.898926 27710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:04.923398 27710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:04.925917 27710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:04.926117 27710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:04.930861 27710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36497
I20260808 02:22:04.930895 27810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36497 every 8 connection(s)
I20260808 02:22:04.931299 27710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:04.932232 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:04.934916 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b: Bootstrap starting.
I20260808 02:22:04.935619 27811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:04.935964 27811 log.cc:824] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:04.936774 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b: No bootstrap required, opened a new log
I20260808 02:22:04.938581 27811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b622094f2d24755adaf5c38490f5b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36497 } }
I20260808 02:22:04.938740 27811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:04.938763 27811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b622094f2d24755adaf5c38490f5b6b, State: Initialized, Role: FOLLOWER
I20260808 02:22:04.938973 27811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b622094f2d24755adaf5c38490f5b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36497 } }
I20260808 02:22:04.939095 27811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:04.939116 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 27710
I20260808 02:22:04.939158 27811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:04.939213 27811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:04.939277 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:04.939527 29382 external_mini_cluster.cc:1428] Setting key 409fc925017292c16d775256e68a13f5
I20260808 02:22:04.939883 27811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b622094f2d24755adaf5c38490f5b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36497 } }
I20260808 02:22:04.940096 27811 leader_election.cc:302] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b622094f2d24755adaf5c38490f5b6b; no voters:
I20260808 02:22:04.940363 27811 leader_election.cc:288] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:04.940393 27817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:04.940593 27817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [term 1 LEADER]: Becoming Leader. State: Replica: 4b622094f2d24755adaf5c38490f5b6b, State: Running, Role: LEADER
I20260808 02:22:04.940742 27811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:04.940757 27817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b622094f2d24755adaf5c38490f5b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36497 } }
I20260808 02:22:04.941304 27818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b622094f2d24755adaf5c38490f5b6b. Latest consensus state: current_term: 1 leader_uuid: "4b622094f2d24755adaf5c38490f5b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b622094f2d24755adaf5c38490f5b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36497 } } }
I20260808 02:22:04.941408 27818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:04.941414 27820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b622094f2d24755adaf5c38490f5b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b622094f2d24755adaf5c38490f5b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36497 } } }
I20260808 02:22:04.941545 27820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:04.941814 27832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:04.942445 27832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:04.944260 27832 catalog_manager.cc:1347] Generated new cluster ID: 51f2bb97efef45aba75cbe02d0a40ecc
I20260808 02:22:04.944314 27832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:04.946529 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:04.940161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:33911 (local address 127.0.0.1:36497)
0808 02:22:04.940397 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:04.940408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:04.940450 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:22:04.940616 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:04.940625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:04.940695 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:04.941203 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:04.941329 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:04.942884 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:04.944019 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:04.944719 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:04.944890 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:04.945724 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:04.945728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:04.945729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:04.945929 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:04.945966 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:04.945984 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:04.946114 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:04.946247 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 02:22:04.946283 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260808 02:22:04.950325 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:04.955075 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:04.951884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:53633 (local address 127.0.0.1:36497)
0808 02:22:04.951928 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:04.951930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:04.952061 (+ 131us) server_negotiation.cc:407] Connection header received
0808 02:22:04.952147 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:04.952148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:04.952160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:04.952211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:04.952360 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:04.952949 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:04.953734 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:04.953967 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:04.954046 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:04.954791 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:04.954793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:04.954793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:04.954798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:04.954814 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:04.954819 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:04.954878 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:04.954953 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 02:22:04.954966 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:04.955453 27832 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:04.956005 27832 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:04.958962 27832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b622094f2d24755adaf5c38490f5b6b: Generated new TSK 0
I20260808 02:22:04.959251 27832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:04.962998 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 27710
W20260808 02:22:04.978199 27688 connection.cc:535] client connection to 127.0.0.1:36497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36497): Connection reset by peer (error 104)
2026-08-08T02:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1559 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:05 dist-test-slave-3h5h krb5kdc[27855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:05 dist-test-slave-3h5h krb5kdc[27855](info): set up 2 sockets
Aug 08 02:22:05 dist-test-slave-3h5h krb5kdc[27855](info): commencing operation
krb5kdc: starting...
W20260808 02:22:06.238843 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:06 dist-test-slave-3h5h krb5kdc[27855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:22:06.380335 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:06.468036 27870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:06.468266 27870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:06.468290 27870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:06.469575 27870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:06.469624 27870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:06.469638 27870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:06.469652 27870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:06.469664 27870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:06.471405 27870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:06.471731 27870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:06.471984 27870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:06.477536 27887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:06.477717 27870 server_base.cc:1034] running on GCE node
W20260808 02:22:06.477581 27882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:06.477535 27881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:06.478191 27870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:06.478525 27870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:06.479714 27870 hybrid_clock.cc:648] HybridClock initialized: now 1786155726479688 us; error 52 us; skew 500 ppm
Aug 08 02:22:06 dist-test-slave-3h5h krb5kdc[27855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:06.487792 27870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:22:06.488421 27870 webserver.cc:460] Webserver started at http://127.0.0.1:39113/ using document root <none> and password file <none>
I20260808 02:22:06.488720 27870 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:06.488803 27870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:06.489018 27870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:06.490218 27870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "1762a7566fe84b108c744373cfaef20c"
format_stamp: "Formatted at 2026-08-08 02:22:06 on dist-test-slave-3h5h"
server_key: "11872025d3073ace7ccc6c7f935a9a28"
server_key_iv: "c5f5b8dd65676ff830a57992c02f65d6"
server_key_version: "encryptionkey@0"
I20260808 02:22:06.490702 27870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "1762a7566fe84b108c744373cfaef20c"
format_stamp: "Formatted at 2026-08-08 02:22:06 on dist-test-slave-3h5h"
server_key: "11872025d3073ace7ccc6c7f935a9a28"
server_key_iv: "c5f5b8dd65676ff830a57992c02f65d6"
server_key_version: "encryptionkey@0"
I20260808 02:22:06.492473 27870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:06.493495 27897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:06.493726 27870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 02:22:06.493851 27870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "1762a7566fe84b108c744373cfaef20c"
format_stamp: "Formatted at 2026-08-08 02:22:06 on dist-test-slave-3h5h"
server_key: "11872025d3073ace7ccc6c7f935a9a28"
server_key_iv: "c5f5b8dd65676ff830a57992c02f65d6"
server_key_version: "encryptionkey@0"
I20260808 02:22:06.493937 27870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:06.497814 27870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:06.500430 27870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:06.500674 27870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:06.505059 27870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42889
I20260808 02:22:06.505090 27978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42889 every 8 connection(s)
I20260808 02:22:06.505473 27870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:06.506548 27979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:06.506894 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 27870
I20260808 02:22:06.507028 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:06.507237 29382 external_mini_cluster.cc:1428] Setting key 3bad0a0ff92d10e456e64655b970b002
I20260808 02:22:06.509950 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c: Bootstrap starting.
I20260808 02:22:06.510176 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:06.507872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:43989 (local address 127.0.0.1:42889)
0808 02:22:06.508118 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:06.508127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:06.508158 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:22:06.508642 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:06.508649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:06.508678 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:06.508817 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:06.509642 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:06.509645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:06.509646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:06.509650 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:06.509693 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:06.509704 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:06.509770 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:06.509866 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 02:22:06.509937 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":116,"threads_started":1}
I20260808 02:22:06.511112 27979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:06.511557 27979 log.cc:824] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:06.511621 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:06.512621 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c: No bootstrap required, opened a new log
I20260808 02:22:06.514536 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:06.513212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:47543 (local address 127.0.0.1:42889)
0808 02:22:06.513279 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:06.513282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:06.513357 (+ 75us) server_negotiation.cc:407] Connection header received
0808 02:22:06.513442 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:06.513443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:06.513448 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:06.513530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:06.514282 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:06.514284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:06.514284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:06.514289 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:06.514305 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:06.514309 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:06.514376 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:06.514444 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:22:06.514461 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:22:06.515686 27979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1762a7566fe84b108c744373cfaef20c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260808 02:22:06.515861 27979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:06.515890 27979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1762a7566fe84b108c744373cfaef20c, State: Initialized, Role: FOLLOWER
I20260808 02:22:06.516032 27979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1762a7566fe84b108c744373cfaef20c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260808 02:22:06.516140 27979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:06.516170 27979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:06.516206 27979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:06.516968 27979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1762a7566fe84b108c744373cfaef20c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260808 02:22:06.517077 27979 leader_election.cc:302] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1762a7566fe84b108c744373cfaef20c; no voters:
I20260808 02:22:06.517259 27979 leader_election.cc:288] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:06.517305 27996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:06.517517 27996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [term 1 LEADER]: Becoming Leader. State: Replica: 1762a7566fe84b108c744373cfaef20c, State: Running, Role: LEADER
I20260808 02:22:06.517601 27979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:06.517722 27996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1762a7566fe84b108c744373cfaef20c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260808 02:22:06.518234 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1762a7566fe84b108c744373cfaef20c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1762a7566fe84b108c744373cfaef20c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } } }
I20260808 02:22:06.518760 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:06.519348 28011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:06.518288 27999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1762a7566fe84b108c744373cfaef20c. Latest consensus state: current_term: 1 leader_uuid: "1762a7566fe84b108c744373cfaef20c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1762a7566fe84b108c744373cfaef20c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } } }
I20260808 02:22:06.519793 27999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:06.519958 28011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:06.521741 28011 catalog_manager.cc:1347] Generated new cluster ID: 25238cfb6474470092bf21d435121146
I20260808 02:22:06.521827 28011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:06.549628 28011 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:06.549880 28011 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:06.558734 28011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1762a7566fe84b108c744373cfaef20c: Generated new TSK 0
I20260808 02:22:06.559010 28011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:06.566632 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 27870
2026-08-08T02:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1600 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:06 dist-test-slave-3h5h krb5kdc[28025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:06 dist-test-slave-3h5h krb5kdc[28025](info): set up 2 sockets
Aug 08 02:22:06 dist-test-slave-3h5h krb5kdc[28025](info): commencing operation
krb5kdc: starting...
W20260808 02:22:07.816344 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:07 dist-test-slave-3h5h krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:07.960098 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:44701
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:44701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:08.061245 28045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:08.061517 28045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:08.061564 28045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:08.062901 28045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:08.062986 28045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:08.063017 28045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:08.063054 28045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:08.063083 28045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:08.064813 28045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:44701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:44701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:08.065218 28045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:08.065487 28045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:08.071204 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:08.071198 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:08.071201 28063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:08.071708 28045 server_base.cc:1034] running on GCE node
I20260808 02:22:08.072153 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:08.072556 28045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:08.073777 28045 hybrid_clock.cc:648] HybridClock initialized: now 1786155728073745 us; error 42 us; skew 500 ppm
Aug 08 02:22:08 dist-test-slave-3h5h krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:08.076781 28045 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:08.077418 28045 webserver.cc:460] Webserver started at http://127.28.177.190:35715/ using document root <none> and password file <none>
I20260808 02:22:08.077731 28045 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:08.077787 28045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:08.077981 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:08.079218 28045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "7158779eebd44c6881d286d135f1aeed"
format_stamp: "Formatted at 2026-08-08 02:22:08 on dist-test-slave-3h5h"
server_key: "1f8d5d1ecbf85268f503b5a1a0898c1b"
server_key_iv: "31cfa3cfba02f206cb382bb60d828090"
server_key_version: "encryptionkey@0"
I20260808 02:22:08.079634 28045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "7158779eebd44c6881d286d135f1aeed"
format_stamp: "Formatted at 2026-08-08 02:22:08 on dist-test-slave-3h5h"
server_key: "1f8d5d1ecbf85268f503b5a1a0898c1b"
server_key_iv: "31cfa3cfba02f206cb382bb60d828090"
server_key_version: "encryptionkey@0"
I20260808 02:22:08.081348 28045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:08.082314 28072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:08.082561 28045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:08.082660 28045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "7158779eebd44c6881d286d135f1aeed"
format_stamp: "Formatted at 2026-08-08 02:22:08 on dist-test-slave-3h5h"
server_key: "1f8d5d1ecbf85268f503b5a1a0898c1b"
server_key_iv: "31cfa3cfba02f206cb382bb60d828090"
server_key_version: "encryptionkey@0"
I20260808 02:22:08.082767 28045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:08.098796 28045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:08.101357 28045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:08.101562 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:08.107091 28045 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:44701
I20260808 02:22:08.107131 28147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:44701 every 8 connection(s)
I20260808 02:22:08.107621 28045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:08.108608 28148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:08.111035 28148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed: Bootstrap starting.
I20260808 02:22:08.111707 28148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:08.112018 28148 log.cc:824] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:08.112941 28148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed: No bootstrap required, opened a new log
I20260808 02:22:08.114841 28148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7158779eebd44c6881d286d135f1aeed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44701 } }
I20260808 02:22:08.114998 28148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:08.115021 28148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7158779eebd44c6881d286d135f1aeed, State: Initialized, Role: FOLLOWER
I20260808 02:22:08.115191 28148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7158779eebd44c6881d286d135f1aeed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44701 } }
I20260808 02:22:08.115257 28148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:08.115279 28148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:08.115334 28148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:08.116302 28148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7158779eebd44c6881d286d135f1aeed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44701 } }
I20260808 02:22:08.116394 28148 leader_election.cc:302] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7158779eebd44c6881d286d135f1aeed; no voters:
I20260808 02:22:08.116581 28148 leader_election.cc:288] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:08.116641 28151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:08.116899 28151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [term 1 LEADER]: Becoming Leader. State: Replica: 7158779eebd44c6881d286d135f1aeed, State: Running, Role: LEADER
I20260808 02:22:08.116915 28148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:08.117184 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 28045
I20260808 02:22:08.117113 28151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7158779eebd44c6881d286d135f1aeed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44701 } }
I20260808 02:22:08.117322 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:08.117561 29382 external_mini_cluster.cc:1428] Setting key 35a77734e1d27842df299f8b8aa3a631
I20260808 02:22:08.117789 28152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7158779eebd44c6881d286d135f1aeed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7158779eebd44c6881d286d135f1aeed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44701 } } }
I20260808 02:22:08.117894 28152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:08.117769 28153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7158779eebd44c6881d286d135f1aeed. Latest consensus state: current_term: 1 leader_uuid: "7158779eebd44c6881d286d135f1aeed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7158779eebd44c6881d286d135f1aeed" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 44701 } } }
I20260808 02:22:08.117951 28153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:08.118430 28166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:08.119031 28166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:08.120872 28166 catalog_manager.cc:1347] Generated new cluster ID: 76b6d05e852d4a3c81a176ec410423f8
I20260808 02:22:08.120929 28166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:08 dist-test-slave-3h5h krb5kdc[28025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:08.129904 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:08.118375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38055 (local address 127.28.177.190:44701)
0808 02:22:08.118703 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:08.118714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:08.119206 (+ 492us) server_negotiation.cc:407] Connection header received
0808 02:22:08.119346 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:08.119354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:08.119419 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:08.119579 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:08.120511 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:08.121752 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:08.122615 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:08.123080 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:08.123237 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:08.125964 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:08.125968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:08.125969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:08.125986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:08.127940 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:08.128445 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:08.128452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:08.128457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:08.128590 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:08.128922 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:08.128923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:08.128924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:08.129217 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:08.129435 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:08.129547 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:22:08.129680 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260808 02:22:08.131021 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:08.139695 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:08.132619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58879 (local address 127.28.177.190:44701)
0808 02:22:08.132714 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:08.132717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:08.132757 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:22:08.132870 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:08.132871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:08.132882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:08.132944 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:08.133531 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:08.134106 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:08.134879 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:08.135140 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:08.135228 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:08.136567 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:08.136569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:08.136569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:08.136578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:08.138193 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:08.138605 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:08.138607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:08.138607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:08.138683 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:08.138938 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:08.138940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:08.138940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:08.139079 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:08.139206 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:08.139390 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 02:22:08.139465 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":70}
I20260808 02:22:08.140071 28166 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:08.140901 28166 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:08.143904 28166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7158779eebd44c6881d286d135f1aeed: Generated new TSK 0
I20260808 02:22:08.144117 28166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:08.148383 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 28045
W20260808 02:22:08.167410 28023 connection.cc:535] client connection to 127.28.177.190:44701 recv error: Network error: failed to read from TLS socket (remote: 127.28.177.190:44701): Connection reset by peer (error 104)
2026-08-08T02:22:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:08 dist-test-slave-3h5h krb5kdc[28188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:08 dist-test-slave-3h5h krb5kdc[28188](info): set up 2 sockets
Aug 08 02:22:08 dist-test-slave-3h5h krb5kdc[28188](info): commencing operation
krb5kdc: starting...
W20260808 02:22:09.439003 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:09 dist-test-slave-3h5h krb5kdc[28188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:09.577131 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42819
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:09.673529 28211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:09.673779 28211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:09.673806 28211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:09.675201 28211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:09.675251 28211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:09.675267 28211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:09.675283 28211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:09.675297 28211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:09.676959 28211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:42819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:09.677189 28211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:09.677382 28211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:09.682964 28220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:09.683065 28211 server_base.cc:1034] running on GCE node
W20260808 02:22:09.682964 28218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:09.682958 28216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:09.683580 28211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:09.683961 28211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:09.685163 28211 hybrid_clock.cc:648] HybridClock initialized: now 1786155729685134 us; error 52 us; skew 500 ppm
Aug 08 02:22:09 dist-test-slave-3h5h krb5kdc[28188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:09.688232 28211 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:09.688782 28211 webserver.cc:460] Webserver started at http://127.28.177.190:38529/ using document root <none> and password file <none>
I20260808 02:22:09.689078 28211 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:09.689159 28211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:09.689327 28211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:09.690533 28211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "7176399854f5425ba9a686f15e900182"
format_stamp: "Formatted at 2026-08-08 02:22:09 on dist-test-slave-3h5h"
server_key: "8c12d1c4e6880b189680391ac6fb3993"
server_key_iv: "7377c4b7ead5132a646e5d27751db1c5"
server_key_version: "encryptionkey@0"
I20260808 02:22:09.690938 28211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "7176399854f5425ba9a686f15e900182"
format_stamp: "Formatted at 2026-08-08 02:22:09 on dist-test-slave-3h5h"
server_key: "8c12d1c4e6880b189680391ac6fb3993"
server_key_iv: "7377c4b7ead5132a646e5d27751db1c5"
server_key_version: "encryptionkey@0"
I20260808 02:22:09.692474 28211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:09.693532 28228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:09.693754 28211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:09.693877 28211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "7176399854f5425ba9a686f15e900182"
format_stamp: "Formatted at 2026-08-08 02:22:09 on dist-test-slave-3h5h"
server_key: "8c12d1c4e6880b189680391ac6fb3993"
server_key_iv: "7377c4b7ead5132a646e5d27751db1c5"
server_key_version: "encryptionkey@0"
I20260808 02:22:09.693961 28211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:09.717954 28211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:09.721321 28211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:09.721513 28211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:09.726763 28211 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:42819
I20260808 02:22:09.726796 28296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:42819 every 8 connection(s)
I20260808 02:22:09.727211 28211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:09.728163 28297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:09.730593 28297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182: Bootstrap starting.
I20260808 02:22:09.731230 28297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:09.731534 28297 log.cc:824] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:09.732295 28297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182: No bootstrap required, opened a new log
I20260808 02:22:09.733946 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 28211
I20260808 02:22:09.734145 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:09.734148 28297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7176399854f5425ba9a686f15e900182" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42819 } }
I20260808 02:22:09.734349 28297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:09.734387 29382 external_mini_cluster.cc:1428] Setting key a638fbeecca22132bcaa1330ecd113b9
I20260808 02:22:09.734402 28297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7176399854f5425ba9a686f15e900182, State: Initialized, Role: FOLLOWER
I20260808 02:22:09.734577 28297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7176399854f5425ba9a686f15e900182" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42819 } }
I20260808 02:22:09.734681 28297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:09.734719 28297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:09.734776 28297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:09.735929 28297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7176399854f5425ba9a686f15e900182" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42819 } }
I20260808 02:22:09.736042 28297 leader_election.cc:302] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7176399854f5425ba9a686f15e900182; no voters:
I20260808 02:22:09.736203 28297 leader_election.cc:288] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:09.736294 28304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:09.736465 28297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:09.736512 28304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [term 1 LEADER]: Becoming Leader. State: Replica: 7176399854f5425ba9a686f15e900182, State: Running, Role: LEADER
I20260808 02:22:09.736704 28304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7176399854f5425ba9a686f15e900182" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42819 } }
I20260808 02:22:09.737164 28305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7176399854f5425ba9a686f15e900182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7176399854f5425ba9a686f15e900182" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42819 } } }
I20260808 02:22:09.737260 28305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:09.737479 28307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7176399854f5425ba9a686f15e900182. Latest consensus state: current_term: 1 leader_uuid: "7176399854f5425ba9a686f15e900182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7176399854f5425ba9a686f15e900182" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 42819 } } }
I20260808 02:22:09.737588 28307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182 [sys.catalog]: This master's current role is: LEADER
W20260808 02:22:09.738500 28329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:09.738582 28329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:09.739208 28319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:09.739413 28319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:09.740980 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:09.734947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35117 (local address 127.28.177.190:42819)
0808 02:22:09.735191 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:09.735201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:09.735238 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:22:09.735381 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:09.735391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:09.735457 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:09.735623 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:09.735838 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:09.737275 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:09.738560 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:09.739020 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:09.739105 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:09.739945 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:09.739949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:09.739950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:09.740192 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:09.740235 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:09.740253 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:09.740353 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:09.740657 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 02:22:09.740691 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260808 02:22:09.741398 28319 catalog_manager.cc:1347] Generated new cluster ID: e7ec20ed3184475fa56a28feaffa91e8
I20260808 02:22:09.741487 28319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:09.741999 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:09.748435 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:09.743483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48289 (local address 127.28.177.190:42819)
0808 02:22:09.743553 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:09.743555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:09.745039 (+ 1484us) server_negotiation.cc:407] Connection header received
0808 02:22:09.745125 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:09.745127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:09.745141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:09.745192 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:09.745340 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:09.746254 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:09.747059 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:09.747308 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:09.747391 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:09.748126 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:09.748128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:09.748128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:09.748135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:09.748158 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:09.748163 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:09.748244 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:09.748312 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:22:09.748327 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:22:09.761022 28319 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:09.761947 28319 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:09.769696 28319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7176399854f5425ba9a686f15e900182: Generated new TSK 0
I20260808 02:22:09.769917 28319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:09.775071 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 28211
W20260808 02:22:09.786367 28185 connection.cc:535] client connection to 127.28.177.190:42819 recv error: Network error: failed to read from TLS socket (remote: 127.28.177.190:42819): Connection reset by peer (error 104)
2026-08-08T02:22:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1618 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:09 dist-test-slave-3h5h krb5kdc[28346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:09 dist-test-slave-3h5h krb5kdc[28346](info): set up 2 sockets
Aug 08 02:22:09 dist-test-slave-3h5h krb5kdc[28346](info): commencing operation
krb5kdc: starting...
W20260808 02:22:11.098718 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:11 dist-test-slave-3h5h krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:11.244112 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:39057
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:11.336912 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:11.337296 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:11.337324 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:11.338590 28365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:11.338639 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:11.338654 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:11.338667 28365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:11.338680 28365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:11.340219 28365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:39057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:11.340433 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:11.340624 28365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:11.346136 28378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:11.346208 28376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:11.346247 28365 server_base.cc:1034] running on GCE node
W20260808 02:22:11.346138 28374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:11.346643 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:11.347010 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:11.348218 28365 hybrid_clock.cc:648] HybridClock initialized: now 1786155731348196 us; error 43 us; skew 500 ppm
Aug 08 02:22:11 dist-test-slave-3h5h krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155731, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:11.351715 28365 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:11.352219 28365 webserver.cc:460] Webserver started at http://127.28.177.190:34823/ using document root <none> and password file <none>
I20260808 02:22:11.352452 28365 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:11.352499 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:11.352604 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:11.353909 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "c5fe5c024dcc48fc97904bcaac512831"
format_stamp: "Formatted at 2026-08-08 02:22:11 on dist-test-slave-3h5h"
server_key: "cd15118b60a6a896f222475b78679ad7"
server_key_iv: "a70ed09dc9b7e055da5b372897efb67b"
server_key_version: "encryptionkey@0"
I20260808 02:22:11.354272 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "c5fe5c024dcc48fc97904bcaac512831"
format_stamp: "Formatted at 2026-08-08 02:22:11 on dist-test-slave-3h5h"
server_key: "cd15118b60a6a896f222475b78679ad7"
server_key_iv: "a70ed09dc9b7e055da5b372897efb67b"
server_key_version: "encryptionkey@0"
I20260808 02:22:11.355863 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 02:22:11.356925 28389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:11.357146 28365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:11.357266 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "c5fe5c024dcc48fc97904bcaac512831"
format_stamp: "Formatted at 2026-08-08 02:22:11 on dist-test-slave-3h5h"
server_key: "cd15118b60a6a896f222475b78679ad7"
server_key_iv: "a70ed09dc9b7e055da5b372897efb67b"
server_key_version: "encryptionkey@0"
I20260808 02:22:11.357343 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:11.361340 28365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:11.364020 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:11.364157 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:11.368960 28365 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:39057
I20260808 02:22:11.369168 28462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:39057 every 8 connection(s)
I20260808 02:22:11.369442 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:11.370697 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:11.370879 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 28365
I20260808 02:22:11.370999 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:11.371239 29382 external_mini_cluster.cc:1428] Setting key e73f3ba14a8c82bcd8086d71524db0fd
I20260808 02:22:11.373807 28463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831: Bootstrap starting.
I20260808 02:22:11.374517 28463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:11.374902 28463 log.cc:824] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:11.375172 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:11.371829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48325 (local address 127.28.177.190:39057)
0808 02:22:11.372989 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:11.373000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:11.373046 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:22:11.373568 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:11.373577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:11.373614 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:11.373797 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:11.374635 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:11.374640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:11.374640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:11.374647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:11.374700 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:11.374717 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:11.374818 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:11.374937 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 02:22:11.374981 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":81,"threads_started":1}
I20260808 02:22:11.375844 28463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831: No bootstrap required, opened a new log
I20260808 02:22:11.376349 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:11.378630 28463 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fe5c024dcc48fc97904bcaac512831" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39057 } }
I20260808 02:22:11.378825 28463 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:11.378912 28463 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5fe5c024dcc48fc97904bcaac512831, State: Initialized, Role: FOLLOWER
I20260808 02:22:11.379065 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:11.377870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37505 (local address 127.28.177.190:39057)
0808 02:22:11.377930 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:11.377933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:11.378072 (+ 139us) server_negotiation.cc:407] Connection header received
0808 02:22:11.378153 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:11.378154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:11.378159 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:11.378205 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:11.378786 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:11.378787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:11.378788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:11.378791 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:11.378803 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:11.378807 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:11.378878 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:11.378951 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 02:22:11.378966 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:22:11.379078 28463 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fe5c024dcc48fc97904bcaac512831" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39057 } }
I20260808 02:22:11.379271 28463 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:11.379313 28463 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:11.379369 28463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:11.380012 28463 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fe5c024dcc48fc97904bcaac512831" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39057 } }
I20260808 02:22:11.380141 28463 leader_election.cc:302] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5fe5c024dcc48fc97904bcaac512831; no voters:
I20260808 02:22:11.380321 28463 leader_election.cc:288] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:11.380362 28481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:11.380558 28481 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [term 1 LEADER]: Becoming Leader. State: Replica: c5fe5c024dcc48fc97904bcaac512831, State: Running, Role: LEADER
I20260808 02:22:11.380649 28463 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:11.380657 28481 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fe5c024dcc48fc97904bcaac512831" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39057 } }
I20260808 02:22:11.381137 28482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5fe5c024dcc48fc97904bcaac512831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fe5c024dcc48fc97904bcaac512831" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39057 } } }
I20260808 02:22:11.381178 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5fe5c024dcc48fc97904bcaac512831. Latest consensus state: current_term: 1 leader_uuid: "c5fe5c024dcc48fc97904bcaac512831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fe5c024dcc48fc97904bcaac512831" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 39057 } } }
I20260808 02:22:11.381306 28482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:11.381322 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:11.381865 28494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:11.382375 28494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:11.384125 28494 catalog_manager.cc:1347] Generated new cluster ID: 0072958a9f0e4113a85e1ccc3318fa60
I20260808 02:22:11.384179 28494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:11.397717 28494 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:11.397929 28494 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:11.401010 28494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5fe5c024dcc48fc97904bcaac512831: Generated new TSK 0
I20260808 02:22:11.401232 28494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:11.408716 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 28365
2026-08-08T02:22:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:11 dist-test-slave-3h5h krb5kdc[28515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:11 dist-test-slave-3h5h krb5kdc[28515](info): set up 2 sockets
Aug 08 02:22:11 dist-test-slave-3h5h krb5kdc[28515](info): commencing operation
krb5kdc: starting...
W20260808 02:22:12.718845 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:12 dist-test-slave-3h5h krb5kdc[28515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:12.860419 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:43291
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:43291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:12.948325 28535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:12.948603 28535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:12.948647 28535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:12.949947 28535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:12.950031 28535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:12.950058 28535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:12.950081 28535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:12.950110 28535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:12.951709 28535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:43291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:43291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:12.952031 28535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:12.952288 28535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:12.957924 28544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:12.957924 28543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:12.957947 28546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:12.958196 28535 server_base.cc:1034] running on GCE node
I20260808 02:22:12.958479 28535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:12.958837 28535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:12.960053 28535 hybrid_clock.cc:648] HybridClock initialized: now 1786155732960008 us; error 57 us; skew 500 ppm
Aug 08 02:22:12 dist-test-slave-3h5h krb5kdc[28515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155732, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:12.963197 28535 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:12.963806 28535 webserver.cc:460] Webserver started at http://127.28.177.190:33379/ using document root <none> and password file <none>
I20260808 02:22:12.964071 28535 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:12.964150 28535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:12.964329 28535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:12.965662 28535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "03c971c1a4524b008b9c3c5a2b2a15ab"
format_stamp: "Formatted at 2026-08-08 02:22:12 on dist-test-slave-3h5h"
server_key: "5bec6c6b45ca34d4234c440131140287"
server_key_iv: "b01a38ee543dd81d1cc97ceefcecffa5"
server_key_version: "encryptionkey@0"
I20260808 02:22:12.966089 28535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "03c971c1a4524b008b9c3c5a2b2a15ab"
format_stamp: "Formatted at 2026-08-08 02:22:12 on dist-test-slave-3h5h"
server_key: "5bec6c6b45ca34d4234c440131140287"
server_key_iv: "b01a38ee543dd81d1cc97ceefcecffa5"
server_key_version: "encryptionkey@0"
I20260808 02:22:12.967701 28535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 02:22:12.968713 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:12.968967 28535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:12.969094 28535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "03c971c1a4524b008b9c3c5a2b2a15ab"
format_stamp: "Formatted at 2026-08-08 02:22:12 on dist-test-slave-3h5h"
server_key: "5bec6c6b45ca34d4234c440131140287"
server_key_iv: "b01a38ee543dd81d1cc97ceefcecffa5"
server_key_version: "encryptionkey@0"
I20260808 02:22:12.969189 28535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:12.981889 28535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:12.984882 28535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:12.985069 28535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:12.990229 28535 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:43291
I20260808 02:22:12.990695 28535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:12.990693 28634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:43291 every 8 connection(s)
I20260808 02:22:12.991549 28635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:12.994313 28635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab: Bootstrap starting.
I20260808 02:22:12.995000 28635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:12.995327 28635 log.cc:824] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:12.996227 28635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab: No bootstrap required, opened a new log
I20260808 02:22:12.997617 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 28535
I20260808 02:22:12.997740 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:12.997937 29382 external_mini_cluster.cc:1428] Setting key 71c646416fe01efe09666e2b1b3e28ad
I20260808 02:22:12.998513 28635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43291 } }
I20260808 02:22:12.998793 28635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:12.998863 28635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c971c1a4524b008b9c3c5a2b2a15ab, State: Initialized, Role: FOLLOWER
I20260808 02:22:12.999006 28635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43291 } }
I20260808 02:22:12.999112 28635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:12.999158 28635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:12.999207 28635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:12.999787 28635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43291 } }
I20260808 02:22:12.999908 28635 leader_election.cc:302] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c971c1a4524b008b9c3c5a2b2a15ab; no voters:
I20260808 02:22:13.000108 28635 leader_election.cc:288] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:13.000141 28642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:13.000254 28642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [term 1 LEADER]: Becoming Leader. State: Replica: 03c971c1a4524b008b9c3c5a2b2a15ab, State: Running, Role: LEADER
I20260808 02:22:13.000403 28635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:13.000411 28642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43291 } }
I20260808 02:22:13.000933 28644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c971c1a4524b008b9c3c5a2b2a15ab. Latest consensus state: current_term: 1 leader_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43291 } } }
I20260808 02:22:13.001060 28644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:13.001294 28645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c971c1a4524b008b9c3c5a2b2a15ab" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 43291 } } }
I20260808 02:22:13.001591 28645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:13.001799 28653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:13.002452 28653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:13.004288 28653 catalog_manager.cc:1347] Generated new cluster ID: dd1122eec1e24f8fa88c5d64de14e269
I20260808 02:22:13.004348 28653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 02:22:13 dist-test-slave-3h5h krb5kdc[28515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:13.012243 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:12.998579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58041 (local address 127.28.177.190:43291)
0808 02:22:12.998811 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:12.998821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:12.998851 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:22:12.999008 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:12.999014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:12.999070 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:12.999257 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:13.000512 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:13.002260 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:13.003263 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:13.003958 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:13.004122 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:13.007339 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:13.007344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:13.007344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:13.007360 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:13.010212 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:13.010679 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:13.010683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:13.010684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:13.010821 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:13.011068 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:13.011069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:13.011069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:13.011416 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:13.011592 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:13.011709 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 02:22:13.011981 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":130,"threads_started":1}
I20260808 02:22:13.013377 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:13.019090 28653 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:13.019866 28653 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:13.025826 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:13.017247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40211 (local address 127.28.177.190:43291)
0808 02:22:13.017306 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:13.017308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:13.017500 (+ 192us) server_negotiation.cc:407] Connection header received
0808 02:22:13.017583 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:13.017584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:13.017599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:13.017660 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:13.018383 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:13.019120 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:13.020232 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:13.020571 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:13.020645 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:13.022167 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:13.022169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:13.022169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:13.022177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:13.023980 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:13.024397 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:13.024398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:13.024399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:13.024464 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:13.024691 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:13.024692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:13.024692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:13.024851 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:13.025138 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:13.025625 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 02:22:13.025723 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:13.026993 28653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c971c1a4524b008b9c3c5a2b2a15ab: Generated new TSK 0
I20260808 02:22:13.027256 28653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:13.034749 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 28535
2026-08-08T02:22:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:13 dist-test-slave-3h5h krb5kdc[28680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:13 dist-test-slave-3h5h krb5kdc[28680](info): set up 2 sockets
Aug 08 02:22:13 dist-test-slave-3h5h krb5kdc[28680](info): commencing operation
krb5kdc: starting...
W20260808 02:22:14.339284 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:14 dist-test-slave-3h5h krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:14.481282 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:33159
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:14.572582 28701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:14.572845 28701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:14.572908 28701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:14.574126 28701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:14.574205 28701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:14.574230 28701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:14.574252 28701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:14.574282 28701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:14.575841 28701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:14.576188 28701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:14.576433 28701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:14.581794 28712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:14.581801 28714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:14.582073 28701 server_base.cc:1034] running on GCE node
W20260808 02:22:14.581807 28711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:14.582423 28701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:14.582741 28701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:14.583923 28701 hybrid_clock.cc:648] HybridClock initialized: now 1786155734583907 us; error 50 us; skew 500 ppm
Aug 08 02:22:14 dist-test-slave-3h5h krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155734, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:14.586741 28701 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:14.587246 28701 webserver.cc:460] Webserver started at http://127.28.177.190:42999/ using document root <none> and password file <none>
I20260808 02:22:14.587525 28701 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:14.587600 28701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:14.587757 28701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:14.588953 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "33bf528aa9504c0aaa3ec23ca515c5b7"
format_stamp: "Formatted at 2026-08-08 02:22:14 on dist-test-slave-3h5h"
server_key: "f5967c9d2d59643a9179459e4725ee62"
server_key_iv: "dbcdf2d5c2f1d8121c0a1837075e7117"
server_key_version: "encryptionkey@0"
I20260808 02:22:14.589327 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "33bf528aa9504c0aaa3ec23ca515c5b7"
format_stamp: "Formatted at 2026-08-08 02:22:14 on dist-test-slave-3h5h"
server_key: "f5967c9d2d59643a9179459e4725ee62"
server_key_iv: "dbcdf2d5c2f1d8121c0a1837075e7117"
server_key_version: "encryptionkey@0"
I20260808 02:22:14.590749 28701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:14.591526 28724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:14.591724 28701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:14.591838 28701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "33bf528aa9504c0aaa3ec23ca515c5b7"
format_stamp: "Formatted at 2026-08-08 02:22:14 on dist-test-slave-3h5h"
server_key: "f5967c9d2d59643a9179459e4725ee62"
server_key_iv: "dbcdf2d5c2f1d8121c0a1837075e7117"
server_key_version: "encryptionkey@0"
I20260808 02:22:14.591915 28701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:14.609313 28701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:14.611670 28701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:14.611882 28701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:14.617091 28701 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:33159
I20260808 02:22:14.617244 28800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:33159 every 8 connection(s)
I20260808 02:22:14.617610 28701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:14.618115 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 28701
I20260808 02:22:14.618249 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:14.618464 29382 external_mini_cluster.cc:1428] Setting key dfbc56b707734e10bb536fb46d0fc448
I20260808 02:22:14.618813 28801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:14.622228 28801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7: Bootstrap starting.
I20260808 02:22:14.623122 28801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:14.623553 28801 log.cc:824] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:14.624745 28801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7: No bootstrap required, opened a new log
I20260808 02:22:14.626121 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:14.619066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57885 (local address 127.28.177.190:33159)
0808 02:22:14.619301 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:14.619310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:14.619339 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:22:14.620094 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:14.620102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:14.620162 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:14.620338 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:14.620599 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:14.621703 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:14.622851 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:14.623591 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:14.623713 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:14.625148 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:14.625153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:14.625154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:14.625399 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:14.625445 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:14.625465 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:14.625558 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:14.625825 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 02:22:14.625916 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260808 02:22:14.627601 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:14.627774 28801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33159 } }
I20260808 02:22:14.627940 28801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:14.627967 28801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bf528aa9504c0aaa3ec23ca515c5b7, State: Initialized, Role: FOLLOWER
I20260808 02:22:14.628125 28801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33159 } }
I20260808 02:22:14.628211 28801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:14.628235 28801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:14.628312 28801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:14.629144 28801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33159 } }
I20260808 02:22:14.629283 28801 leader_election.cc:302] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33bf528aa9504c0aaa3ec23ca515c5b7; no voters:
I20260808 02:22:14.629560 28801 leader_election.cc:288] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:14.629585 28815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:14.629710 28815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [term 1 LEADER]: Becoming Leader. State: Replica: 33bf528aa9504c0aaa3ec23ca515c5b7, State: Running, Role: LEADER
I20260808 02:22:14.629803 28815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33159 } }
I20260808 02:22:14.629849 28801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:14.630200 28815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33159 } } }
I20260808 02:22:14.630365 28815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:14.630582 28815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33bf528aa9504c0aaa3ec23ca515c5b7. Latest consensus state: current_term: 1 leader_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bf528aa9504c0aaa3ec23ca515c5b7" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 33159 } } }
I20260808 02:22:14.630708 28815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:14.631487 28825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:14.632367 28825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:14.633219 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:14.629366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48105 (local address 127.28.177.190:33159)
0808 02:22:14.629415 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:14.629416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:14.629572 (+ 156us) server_negotiation.cc:407] Connection header received
0808 02:22:14.629668 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:14.629669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:14.629683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:14.629714 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:14.629958 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:14.630692 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:14.631880 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:14.632109 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:14.632192 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:14.632942 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:14.632943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:14.632944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:14.632949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:14.632965 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:14.632969 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:14.633024 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:14.633116 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 02:22:14.633149 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:14.634164 28825 catalog_manager.cc:1347] Generated new cluster ID: ea410c859d1d4740900aa87a1a94ddef
I20260808 02:22:14.634208 28825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:14.650141 28825 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:14.651003 28825 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:14.654800 28825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33bf528aa9504c0aaa3ec23ca515c5b7: Generated new TSK 0
I20260808 02:22:14.655010 28825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:14.661823 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 28701
2026-08-08T02:22:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:14 dist-test-slave-3h5h krb5kdc[28846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:14 dist-test-slave-3h5h krb5kdc[28846](info): set up 2 sockets
Aug 08 02:22:14 dist-test-slave-3h5h krb5kdc[28846](info): commencing operation
krb5kdc: starting...
W20260808 02:22:15.909233 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:15 dist-test-slave-3h5h krb5kdc[28846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:16.045289 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41589
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:16.132302 28863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:16.132592 28863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:16.132635 28863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:16.133869 28863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:16.133962 28863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:16.133991 28863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:16.134022 28863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:16.134059 28863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:16.135591 28863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:16.135881 28863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:16.136137 28863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:16.141562 28873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:16.141757 28870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:16.141772 28875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:16.141902 28863 server_base.cc:1034] running on GCE node
I20260808 02:22:16.142205 28863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:16.142556 28863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:16.143755 28863 hybrid_clock.cc:648] HybridClock initialized: now 1786155736143729 us; error 47 us; skew 500 ppm
Aug 08 02:22:16 dist-test-slave-3h5h krb5kdc[28846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155736, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:16.146652 28863 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:16.147169 28863 webserver.cc:460] Webserver started at http://127.28.177.190:33635/ using document root <none> and password file <none>
I20260808 02:22:16.147423 28863 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:16.147501 28863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:16.147655 28863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:16.148924 28863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211"
format_stamp: "Formatted at 2026-08-08 02:22:16 on dist-test-slave-3h5h"
server_key: "1762cfec7597c91db9851da660ef4a35"
server_key_iv: "064e8f0581611c61c9548c71dea0bd52"
server_key_version: "encryptionkey@0"
I20260808 02:22:16.149323 28863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211"
format_stamp: "Formatted at 2026-08-08 02:22:16 on dist-test-slave-3h5h"
server_key: "1762cfec7597c91db9851da660ef4a35"
server_key_iv: "064e8f0581611c61c9548c71dea0bd52"
server_key_version: "encryptionkey@0"
I20260808 02:22:16.150782 28863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 02:22:16.151789 28883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:16.152172 28863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:16.152330 28863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211"
format_stamp: "Formatted at 2026-08-08 02:22:16 on dist-test-slave-3h5h"
server_key: "1762cfec7597c91db9851da660ef4a35"
server_key_iv: "064e8f0581611c61c9548c71dea0bd52"
server_key_version: "encryptionkey@0"
I20260808 02:22:16.152424 28863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:16.156675 28863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:16.159091 28863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:16.159298 28863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:16.164471 28863 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:41589
I20260808 02:22:16.164521 28964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:41589 every 8 connection(s)
I20260808 02:22:16.164999 28863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:16.166188 28965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:16.168649 28965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211: Bootstrap starting.
I20260808 02:22:16.169332 28965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:16.169651 28965 log.cc:824] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:16.170558 28965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211: No bootstrap required, opened a new log
I20260808 02:22:16.171686 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 28863
I20260808 02:22:16.171820 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:16.172056 29382 external_mini_cluster.cc:1428] Setting key 3d48e5c65fbde33793af378c4ac5601f
I20260808 02:22:16.173178 28965 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41589 } }
I20260808 02:22:16.173357 28965 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:22:16.173425 28965 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b4381e8dab4d12b7e5cf6ed0ee4211, State: Initialized, Role: FOLLOWER
I20260808 02:22:16.173630 28965 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41589 } }
I20260808 02:22:16.173777 28965 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:22:16.173822 28965 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:22:16.173878 28965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:16.174607 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:16.172665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58727 (local address 127.28.177.190:41589)
0808 02:22:16.172895 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:16.172905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:16.172942 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:22:16.173155 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:16.173162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:16.173190 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:16.173345 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:16.174125 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:16.174137 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:16.174137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:16.174141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:16.174181 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:16.174192 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:16.174266 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:16.174347 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 02:22:16.174390 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260808 02:22:16.175064 28965 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41589 } }
I20260808 02:22:16.175203 28965 leader_election.cc:302] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b4381e8dab4d12b7e5cf6ed0ee4211; no voters:
I20260808 02:22:16.175411 28965 leader_election.cc:288] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:22:16.175454 28970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:16.175675 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:16.175701 28965 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:16.175789 28970 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [term 1 LEADER]: Becoming Leader. State: Replica: e5b4381e8dab4d12b7e5cf6ed0ee4211, State: Running, Role: LEADER
I20260808 02:22:16.175874 28970 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41589 } }
I20260808 02:22:16.176219 28971 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41589 } } }
I20260808 02:22:16.176306 28971 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:16.176502 28972 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b4381e8dab4d12b7e5cf6ed0ee4211. Latest consensus state: current_term: 1 leader_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b4381e8dab4d12b7e5cf6ed0ee4211" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41589 } } }
I20260808 02:22:16.176579 28972 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211 [sys.catalog]: This master's current role is: LEADER
W20260808 02:22:16.177826 29004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:16.177901 29004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:16.177975 28986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:16.178176 28986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:16.179651 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:16.177795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49211 (local address 127.28.177.190:41589)
0808 02:22:16.177830 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:16.177832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:16.178093 (+ 261us) server_negotiation.cc:407] Connection header received
0808 02:22:16.178212 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:16.178213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:16.178220 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:16.178276 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:16.179406 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:16.179407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:16.179408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:22:16.179413 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:16.179428 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:22:16.179433 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:16.179496 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:16.179575 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 02:22:16.179591 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 02:22:16.179996 28986 catalog_manager.cc:1347] Generated new cluster ID: 6479c4a54a7547109b64bf59a2c7498f
I20260808 02:22:16.180091 28986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:16.202847 28986 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:16.203167 28986 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:16.207415 28986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5b4381e8dab4d12b7e5cf6ed0ee4211: Generated new TSK 0
I20260808 02:22:16.207659 28986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:22:16.210673 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 28863
2026-08-08T02:22:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1550 ms)
[----------] 18 tests from AuthTokenIssuingTest (29109 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:16 dist-test-slave-3h5h krb5kdc[29015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:16 dist-test-slave-3h5h krb5kdc[29015](info): set up 2 sockets
Aug 08 02:22:16 dist-test-slave-3h5h krb5kdc[29015](info): commencing operation
krb5kdc: starting...
W20260808 02:22:17.555430 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:17 dist-test-slave-3h5h krb5kdc[29015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:17.706071 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:34673
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:34673,127.28.177.189:37331,127.28.177.188:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:17.794056 29034 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:22:17.794325 29034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:17.794358 29034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:17.794401 29034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:17.795607 29034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:17.795683 29034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:17.795707 29034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:17.795730 29034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:17.795786 29034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:17.797539 29034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:34673,127.28.177.189:37331,127.28.177.188:34535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:17.797832 29034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:17.798082 29034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:17.803743 29045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:17.803709 29040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:17.803709 29042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:17.803728 29034 server_base.cc:1034] running on GCE node
I20260808 02:22:17.804504 29034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:17.804898 29034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:17.806146 29034 hybrid_clock.cc:648] HybridClock initialized: now 1786155737806110 us; error 52 us; skew 500 ppm
Aug 08 02:22:17 dist-test-slave-3h5h krb5kdc[29015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:17.815551 29034 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:17.816120 29034 webserver.cc:460] Webserver started at http://127.28.177.190:45803/ using document root <none> and password file <none>
I20260808 02:22:17.816394 29034 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:17.816469 29034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:17.816630 29034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:17.817904 29034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "5f24343420014bc2b269b4ed910f9a3a"
format_stamp: "Formatted at 2026-08-08 02:22:17 on dist-test-slave-3h5h"
server_key: "b03e7719c26899e94b4a3d2d50d3582e"
server_key_iv: "5ee864e07e38d6215fbeeed286ade11d"
server_key_version: "encryptionkey@0"
I20260808 02:22:17.818310 29034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "5f24343420014bc2b269b4ed910f9a3a"
format_stamp: "Formatted at 2026-08-08 02:22:17 on dist-test-slave-3h5h"
server_key: "b03e7719c26899e94b4a3d2d50d3582e"
server_key_iv: "5ee864e07e38d6215fbeeed286ade11d"
server_key_version: "encryptionkey@0"
I20260808 02:22:17.819820 29034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:17.820706 29056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:17.821017 29034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:17.821135 29034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "5f24343420014bc2b269b4ed910f9a3a"
format_stamp: "Formatted at 2026-08-08 02:22:17 on dist-test-slave-3h5h"
server_key: "b03e7719c26899e94b4a3d2d50d3582e"
server_key_iv: "5ee864e07e38d6215fbeeed286ade11d"
server_key_version: "encryptionkey@0"
I20260808 02:22:17.821235 29034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:17.848542 29034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:17.851586 29034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:17.851800 29034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:17.856666 29034 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:34673
I20260808 02:22:17.856765 29126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:34673 every 8 connection(s)
I20260808 02:22:17.857224 29034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:17.858342 29127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:17.860086 29127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:17.863867 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29034
I20260808 02:22:17.864166 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:17.864463 29382 external_mini_cluster.cc:1428] Setting key 9a145d33e842b3c3616017077af97204
Aug 08 02:22:17 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:17.872522 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.860938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34673 (local address 127.0.0.1:46732)
0808 02:22:17.861192 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.861247 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:17.861891 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:17.862237 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:17.862240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:17.862586 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:17.862770 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:17.862774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:17.864000 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:17.864001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:17.865043 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:17.865045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:17.865628 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:17.865629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:17.865706 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:17.866471 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:17.866486 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:17.868482 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:17.870707 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:17.870715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:17.870718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:17.870973 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:17.871267 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:17.871268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:17.871268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:17.871374 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:17.871800 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:17.871806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:17.871881 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 02:22:17.872010 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 02:22:17.872306 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":119,"thread_start_us":124,"threads_started":1}
I20260808 02:22:17.872522 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.861030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.28.177.190:34673)
0808 02:22:17.861205 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:17.861216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:17.861369 (+ 153us) server_negotiation.cc:407] Connection header received
0808 02:22:17.862084 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:17.862087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:17.862116 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:17.862231 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:17.862901 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:17.863882 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:17.865155 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:17.865513 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:17.865664 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:17.868612 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:17.868622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:17.868622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:17.868631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:17.870581 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:17.871085 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:17.871089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:17.871089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:17.871188 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:17.871449 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:17.871450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:17.871451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:17.871711 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:17.871813 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:17.872053 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 02:22:17.872231 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260808 02:22:17.873492 29127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:17.873999 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.873852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39498)
0808 02:22:17.873914 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.873973 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 02:22:17.874220 29058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
I20260808 02:22:17.874775 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.874680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39500)
0808 02:22:17.874728 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.874757 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.28.177.189@KRBTEST.COM; defaulting to no policy
W20260808 02:22:17.874918 29127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:37331: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab.
Entry for principal kudu/127.28.177.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab.
I20260808 02:22:17.897404 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:37331
--webserver_interface=127.28.177.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab
--principal=kudu/127.28.177.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:34673,127.28.177.189:37331,127.28.177.188:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:22:17.924170 29127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } attempt: 1
I20260808 02:22:17.924917 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.924701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39514)
0808 02:22:17.924786 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.924873 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:22:17.925551 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.925412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39526)
0808 02:22:17.925473 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.925512 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 02:22:17.925704 29127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:37331: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
W20260808 02:22:17.987967 29136 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:22:17.988272 29136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:17.988304 29136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:17.988349 29136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:17.989591 29136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:17.989668 29136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:17.989693 29136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:17.989715 29136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:17.989744 29136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:17.991359 29136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:34673,127.28.177.189:37331,127.28.177.188:34535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab
--principal=kudu/127.28.177.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.177.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:17.991674 29136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:17.991916 29136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:22:17.993870 29127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } attempt: 2
I20260808 02:22:17.994380 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.994238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39536)
0808 02:22:17.994310 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.994352 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:22:17.994768 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:17.994639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39538)
0808 02:22:17.994724 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:17.994749 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260808 02:22:17.994858 29127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:37331: Network error: Client connection negotiation failed: client connection to 127.28.177.189:37331: connect: Connection refused (error 111)
W20260808 02:22:17.997824 29150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:17.997836 29143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:17.997836 29146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:17.998116 29136 server_base.cc:1034] running on GCE node
I20260808 02:22:17.998389 29136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:17.998677 29136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:17.999872 29136 hybrid_clock.cc:648] HybridClock initialized: now 1786155737999858 us; error 52 us; skew 500 ppm
Aug 08 02:22:17 dist-test-slave-3h5h krb5kdc[29015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:18.002851 29136 init.cc:377] Logged in from keytab as kudu/127.28.177.189@KRBTEST.COM (short username kudu)
I20260808 02:22:18.003460 29136 webserver.cc:460] Webserver started at http://127.28.177.189:39497/ using document root <none> and password file <none>
I20260808 02:22:18.003744 29136 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:18.003818 29136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:18.003978 29136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:18.005209 29136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data/instance:
uuid: "90c9bf170eaa4f6e92c51d374b4f4055"
format_stamp: "Formatted at 2026-08-08 02:22:17 on dist-test-slave-3h5h"
server_key: "24b1bc2b0814f7623ac2ef7cd3a4cf8c"
server_key_iv: "d69f5b1624fa8e116137511903ce4e40"
server_key_version: "encryptionkey@0"
I20260808 02:22:18.005614 29136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal/instance:
uuid: "90c9bf170eaa4f6e92c51d374b4f4055"
format_stamp: "Formatted at 2026-08-08 02:22:17 on dist-test-slave-3h5h"
server_key: "24b1bc2b0814f7623ac2ef7cd3a4cf8c"
server_key_iv: "d69f5b1624fa8e116137511903ce4e40"
server_key_version: "encryptionkey@0"
I20260808 02:22:18.007123 29136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:22:18.008126 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:18.008348 29136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:18.008473 29136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal
uuid: "90c9bf170eaa4f6e92c51d374b4f4055"
format_stamp: "Formatted at 2026-08-08 02:22:17 on dist-test-slave-3h5h"
server_key: "24b1bc2b0814f7623ac2ef7cd3a4cf8c"
server_key_iv: "d69f5b1624fa8e116137511903ce4e40"
server_key_version: "encryptionkey@0"
I20260808 02:22:18.008562 29136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:18.020550 29136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:18.023278 29136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:18.023466 29136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:18.028287 29136 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.189:37331
I20260808 02:22:18.028332 29239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.189:37331 every 8 connection(s)
I20260808 02:22:18.028961 29136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
I20260808 02:22:18.029776 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:18.031476 29240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:18.033891 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29136
I20260808 02:22:18.034029 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-1/wal/instance
I20260808 02:22:18.034343 29382 external_mini_cluster.cc:1428] Setting key 0e9b9601223edd4810e8c556f98ee5a6
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:18.042641 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.032392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.28.177.190:34673)
0808 02:22:18.032445 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.032447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.032521 (+ 74us) server_negotiation.cc:407] Connection header received
0808 02:22:18.033126 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.033127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.033140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.033211 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.034045 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.034846 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.035945 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.036249 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.036310 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.039172 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.039174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.039175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.039185 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.041123 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.041557 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.041558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.041559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.041627 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.041888 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.041889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.041890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.042045 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.042128 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.042420 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.042517 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:18.042764 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.032219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34673 (local address 127.0.0.1:46742)
0808 02:22:18.032396 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.032443 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.033025 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.033313 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.033317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.033661 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.033889 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.033893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.034959 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.034960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.035848 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.035850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.036341 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.036342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.036500 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.037283 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.037296 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.039046 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.041220 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.041228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.041230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.041470 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.041701 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.041701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.041702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.041818 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.042131 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.042140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.042274 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.042367 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.042500 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260808 02:22:18.043516 29240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.177.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.188@KRBTEST.COM" created.
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:22:18.052826 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.043902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39554)
0808 02:22:18.044070 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.044081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.044152 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.044363 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.044364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.044525 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.044579 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.044580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.045602 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.045604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.046186 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.046188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.046749 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.046751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.046826 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.047327 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.047338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.049006 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.051240 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.051241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.051242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.051496 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.051769 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.051770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.051771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.051851 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.052505 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.052506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.052548 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.052611 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.052724 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 02:22:18.053819 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.044002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.28.177.189:37331)
0808 02:22:18.044193 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.044199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.044226 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:22:18.044255 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.044257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.044268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.044364 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.044666 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.045503 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.046296 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.046651 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.046741 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.049137 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.049149 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.049150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.049160 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.051160 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.051612 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.051617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.051617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.051686 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.051919 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.051920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.051920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.052379 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.052479 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.053561 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.053691 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260808 02:22:18.054911 29240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:18.055459 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.055292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40930)
0808 02:22:18.055355 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.055425 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 02:22:18.055694 29163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
I20260808 02:22:18.057379 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.057212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40936)
0808 02:22:18.057304 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.057342 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 02:22:18.060743 29240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:34535: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab.
Entry for principal kudu/127.28.177.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab.
I20260808 02:22:18.069372 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:34535
--webserver_interface=127.28.177.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab
--principal=kudu/127.28.177.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:34673,127.28.177.189:37331,127.28.177.188:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:22:18.085999 29127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } attempt: 3
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:22:18.093521 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.086464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.28.177.189:37331)
0808 02:22:18.086542 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.086544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.086556 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:22:18.086589 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.086590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.086604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.086698 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.087069 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.087583 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.088396 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.088626 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.088734 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.090887 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.090888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.090889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.090898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.092288 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.092664 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.092665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.092666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.092726 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.092962 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.092963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.092964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.093085 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.093165 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.093330 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.093417 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:22:18.093521 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.086372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39564)
0808 02:22:18.086440 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.086456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.086526 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.086711 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.086712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.086883 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.086966 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.086968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.087751 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.087753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.088304 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.088306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.088738 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.088740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.088823 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.089341 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.089351 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.090792 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.092384 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.092385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.092385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.092568 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.092797 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.092798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.092798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.092882 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.093162 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.093163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.093213 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.093304 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.093415 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 02:22:18.094187 29127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:18.094558 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.094451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40942)
0808 02:22:18.094500 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.094534 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:22:18.094847 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.094792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40948)
0808 02:22:18.094815 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.094834 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260808 02:22:18.094962 29127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:34535: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
I20260808 02:22:18.110052 29240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } attempt: 1
I20260808 02:22:18.110741 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.110532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40962)
0808 02:22:18.110626 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.110705 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 02:22:18.111322 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.111204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40970)
0808 02:22:18.111263 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.111298 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 02:22:18.111435 29240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:34535: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
I20260808 02:22:18.126112 29127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } attempt: 1
I20260808 02:22:18.126717 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.126563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40976)
0808 02:22:18.126639 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.126684 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 02:22:18.127100 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.126980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:40986)
0808 02:22:18.127029 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.127061 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 02:22:18.127204 29127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:34535: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
W20260808 02:22:18.162947 29249 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:22:18.163184 29249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:18.163203 29249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:18.163224 29249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:18.164448 29249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:18.164494 29249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:18.164510 29249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:18.164521 29249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:18.164533 29249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:18.166188 29249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:34673,127.28.177.189:37331,127.28.177.188:34535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab
--principal=kudu/127.28.177.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.177.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:18.166456 29249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:18.166711 29249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:18.172173 29257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:18.172283 29249 server_base.cc:1034] running on GCE node
W20260808 02:22:18.172312 29256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:18.172243 29259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:18.172814 29249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:18.173243 29249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:18.174460 29249 hybrid_clock.cc:648] HybridClock initialized: now 1786155738174443 us; error 56 us; skew 500 ppm
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:18.177604 29249 init.cc:377] Logged in from keytab as kudu/127.28.177.188@KRBTEST.COM (short username kudu)
I20260808 02:22:18.178249 29249 webserver.cc:460] Webserver started at http://127.28.177.188:42131/ using document root <none> and password file <none>
I20260808 02:22:18.178555 29249 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:18.178638 29249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:18.178817 29249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:18.179592 29240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } attempt: 2
I20260808 02:22:18.180186 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.179919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41002)
0808 02:22:18.180109 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.180155 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 02:22:18.180171 29249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data/instance:
uuid: "fba2559e648245d7bc66b4b704be1fb3"
format_stamp: "Formatted at 2026-08-08 02:22:18 on dist-test-slave-3h5h"
server_key: "fb4b8f493a033f3f20f88b47badb98f3"
server_key_iv: "a2bd00fd0b83c419e0a700982280dda4"
server_key_version: "encryptionkey@0"
I20260808 02:22:18.180564 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.180467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41010)
0808 02:22:18.180514 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.180539 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 02:22:18.180673 29240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:34535: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
I20260808 02:22:18.180714 29249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal/instance:
uuid: "fba2559e648245d7bc66b4b704be1fb3"
format_stamp: "Formatted at 2026-08-08 02:22:18 on dist-test-slave-3h5h"
server_key: "fb4b8f493a033f3f20f88b47badb98f3"
server_key_iv: "a2bd00fd0b83c419e0a700982280dda4"
server_key_version: "encryptionkey@0"
I20260808 02:22:18.182435 29249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 02:22:18.183336 29269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:18.183609 29249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:18.183719 29249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal
uuid: "fba2559e648245d7bc66b4b704be1fb3"
format_stamp: "Formatted at 2026-08-08 02:22:18 on dist-test-slave-3h5h"
server_key: "fb4b8f493a033f3f20f88b47badb98f3"
server_key_iv: "a2bd00fd0b83c419e0a700982280dda4"
server_key_version: "encryptionkey@0"
I20260808 02:22:18.183781 29249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:18.197129 29249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:18.199631 29249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:18.199787 29249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:18.202380 29127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } attempt: 2
I20260808 02:22:18.202931 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.202737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41014)
0808 02:22:18.202822 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.202891 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 02:22:18.203516 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.203374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41030)
0808 02:22:18.203434 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.203465 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 02:22:18.203639 29127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:34535: Network error: Client connection negotiation failed: client connection to 127.28.177.188:34535: connect: Connection refused (error 111)
I20260808 02:22:18.204521 29249 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.188:34535
I20260808 02:22:18.204567 29332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.188:34535 every 8 connection(s)
I20260808 02:22:18.205027 29249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
I20260808 02:22:18.205785 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29249
I20260808 02:22:18.205901 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155444117209-29382-0/minicluster-data/master-2/wal/instance
I20260808 02:22:18.206087 29382 external_mini_cluster.cc:1428] Setting key d161a563102915150ad2a16d90f1b2d9
I20260808 02:22:18.206184 29333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:18.208784 29333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:18.219178 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.209604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.28.177.190:34673)
0808 02:22:18.209659 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.209676 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.209915 (+ 239us) server_negotiation.cc:407] Connection header received
0808 02:22:18.209986 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.209987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.209998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.210034 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.210707 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.211490 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.212953 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.213138 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.213198 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.215621 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.215623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.215623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.215633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.217357 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.217939 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.217940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.217940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.218017 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.218312 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.218313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.218313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.218480 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.218561 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.218805 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.218921 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:18.219353 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.209535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:34673 (local address 127.0.0.1:46750)
0808 02:22:18.209772 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.209822 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.209915 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.210105 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.210108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.210481 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.210574 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.210576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.211751 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.211753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.212821 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.212825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.213240 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.213242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.213355 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.213855 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.213864 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.215481 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.217469 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.217478 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.217480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.217827 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.218115 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.218116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.218116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.218235 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.218554 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.218560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.218621 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.218755 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.218954 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260808 02:22:18.219696 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.206754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.28.177.188:34535)
0808 02:22:18.206971 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.206978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.207005 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:22:18.207493 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.207500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.207547 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.207709 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.208406 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.209606 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.210824 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.211563 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.211668 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.214998 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.215006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.215007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.215014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.217399 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.217834 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.217839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.217839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.218026 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.218292 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.218293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.218294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.218624 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.218809 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.218859 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.219069 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260808 02:22:18.220381 29333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:18.230031 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.220709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:37331 (local address 127.0.0.1:39568)
0808 02:22:18.220858 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.220871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.220924 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.221080 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.221080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.221306 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.221377 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.221380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.222563 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.222564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.223557 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.223560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.224180 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.224181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.224246 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.224942 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.224947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.226477 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.228775 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.228776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.228777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.229033 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.229249 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.229249 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.229268 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.229344 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.229684 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.229685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.229735 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.229814 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.229916 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 02:22:18.230199 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.220868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.28.177.190:34673)
0808 02:22:18.220911 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.220913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.220953 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:22:18.221015 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.221016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.221025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.221091 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.221685 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.222445 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.223640 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.223987 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.224059 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.227251 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.227254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.227254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.227263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.228851 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.229295 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.229297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.229297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.229350 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.229628 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.229629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.229629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.229787 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.229984 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.230023 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.230105 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 02:22:18.230244 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.220783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.28.177.189:37331)
0808 02:22:18.220863 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.220866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.220908 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:22:18.220983 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.220985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.220997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.221075 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.221496 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.222428 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.223655 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.224045 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.224112 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.226743 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.226745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.226746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.226753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.228672 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.229104 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.229106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.229106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.229170 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.229434 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.229435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.229435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.229609 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.229681 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.229875 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.230109 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:22:18.230718 29333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:22:18.238112 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.231066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41046)
0808 02:22:18.231227 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.231238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.231288 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.231414 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.231415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.231599 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.231639 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.231642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.232466 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.232467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.233073 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.233076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.233483 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.233484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.233531 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.233983 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.233987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.235229 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.237079 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.237079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.237080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.237247 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.237458 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.237459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.237459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.237512 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.237766 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.237766 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.237806 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.237889 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.237987 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260808 02:22:18.238117 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.231182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.28.177.188:34535)
0808 02:22:18.231225 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.231227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.231273 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:22:18.231345 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.231346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.231355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.231399 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.231718 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.232380 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.233177 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.233398 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.233492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.235313 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.235315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.235315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.235322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.236987 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.237331 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.237333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.237333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.237383 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.237581 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.237582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.237583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.237698 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.237773 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.237910 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.238037 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":22}
I20260808 02:22:18.239379 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.231520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.28.177.189:37331)
0808 02:22:18.231558 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.231559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.231569 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:18.231589 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.231590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.231599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.231654 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.232135 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.232765 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.233557 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.233916 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.233986 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.236679 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.236680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.236680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.236687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.238121 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.238533 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.238534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.238535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.238592 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.238829 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.238830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.238830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.238937 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.239163 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.239204 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.239278 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 02:22:18.239907 29333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: Bootstrap starting.
I20260808 02:22:18.241014 29333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:18.241384 29333 log.cc:824] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:18.241539 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:18.242230 29333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: No bootstrap required, opened a new log
I20260808 02:22:18.242659 29095 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5f24343420014bc2b269b4ed910f9a3a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46754
I20260808 02:22:18.243939 29333 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.244067 29333 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba2559e648245d7bc66b4b704be1fb3, State: Initialized, Role: FOLLOWER
I20260808 02:22:18.244160 29333 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.244503 29333 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:18.244480 29346 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.244632 29346 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:22:18.246559 29369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:18.246641 29369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:18.246836 29249 master_runner.cc:186] Error getting master registration for 127.28.177.190:34673: OK, instance_id { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" instance_seqno: 1786155737855253 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:22:18.249804 29249 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:22:18.271826 29240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } attempt: 3
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:22:18.281224 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.272309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41058)
0808 02:22:18.272392 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.272408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.272503 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.272720 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.272722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.272960 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.273021 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.273024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.274095 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.274097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.274957 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.274960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.275534 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.275535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.275604 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.276188 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.276194 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.277823 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.279926 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.279927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.279928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.280166 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.280397 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.280398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.280398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.280474 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.280864 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.280865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.280918 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.280996 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.281106 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 02:22:18.281244 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.272373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.28.177.188:34535)
0808 02:22:18.272435 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.272437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.272502 (+ 65us) server_negotiation.cc:407] Connection header received
0808 02:22:18.272592 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.272593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.272608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.272682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.273122 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.273985 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.275093 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.275435 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.275522 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.277928 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.277930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.277931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.277940 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.279831 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.280256 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.280257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.280257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.280323 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.280545 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.280546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.280546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.280769 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.280867 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.281038 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.281132 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:22:18.283080 29240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: Bootstrap starting.
I20260808 02:22:18.283751 29240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:18.284039 29240 log.cc:824] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:18.284955 29240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: No bootstrap required, opened a new log
I20260808 02:22:18.286670 29240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.286865 29240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c9bf170eaa4f6e92c51d374b4f4055, State: Initialized, Role: FOLLOWER
I20260808 02:22:18.287042 29240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.287351 29372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.287452 29240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:18.287473 29372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:22:18.289592 29393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:18.289660 29393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:18.289836 29136 master_runner.cc:186] Error getting master registration for 127.28.177.190:34673: OK, instance_id { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" instance_seqno: 1786155737855253 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:22:18.293069 29136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:22:18.302816 29127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } attempt: 3
Aug 08 02:22:18 dist-test-slave-3h5h krb5kdc[29015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:22:18.313031 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.303226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:34535 (local address 127.0.0.1:41070)
0808 02:22:18.303293 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:18.303307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:18.303358 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:18.303539 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:18.303540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:18.303750 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:18.303813 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.303815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.305623 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.305625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.306217 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:18.306219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.306769 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.306771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:18.306845 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.307375 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:18.307426 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:18.309101 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:18.311419 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.311420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.311421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.311736 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.312020 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:18.312021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:18.312022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:18.312223 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:18.312547 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:18.312549 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:18.312619 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 02:22:18.312728 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:22:18.312881 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 02:22:18.313655 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.303303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.28.177.188:34535)
0808 02:22:18.303351 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.303353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.303365 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:22:18.303452 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.303453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.303467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.303524 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.303917 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.305530 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.306324 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.306672 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.306766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.309194 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.309195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.309196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.309205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.311323 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.311823 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.311825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.311826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.311897 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.312309 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.312310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.312311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.312475 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.312547 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.313449 (+ 902us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.313545 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 02:22:18.315210 29127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a: Bootstrap starting.
I20260808 02:22:18.315963 29127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:18.316318 29127 log.cc:824] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:18.317301 29127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a: No bootstrap required, opened a new log
I20260808 02:22:18.318889 29127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.319056 29127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24343420014bc2b269b4ed910f9a3a, State: Initialized, Role: FOLLOWER
I20260808 02:22:18.319161 29127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.319547 29127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:18.319599 29095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:22:18.319648 29095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:18.320683 29095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.321056 29095 leader_election.cc:288] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [CANDIDATE]: Term 1 election: Requested vote from peers 90c9bf170eaa4f6e92c51d374b4f4055 (127.28.177.189:37331), fba2559e648245d7bc66b4b704be1fb3 (127.28.177.188:34535)
I20260808 02:22:18.321982 29200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f24343420014bc2b269b4ed910f9a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "90c9bf170eaa4f6e92c51d374b4f4055"
I20260808 02:22:18.322166 29200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [term 0 FOLLOWER]: Advancing to term 1
W20260808 02:22:18.323105 29410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:18.323163 29410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:18.323269 29200 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24343420014bc2b269b4ed910f9a3a in term 1.
I20260808 02:22:18.323570 29299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f24343420014bc2b269b4ed910f9a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fba2559e648245d7bc66b4b704be1fb3"
I20260808 02:22:18.323711 29299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:18.324810 29299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24343420014bc2b269b4ed910f9a3a in term 1.
I20260808 02:22:18.325663 29058 leader_election.cc:302] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24343420014bc2b269b4ed910f9a3a, 90c9bf170eaa4f6e92c51d374b4f4055; no voters:
I20260808 02:22:18.328011 29034 master_runner.cc:263] No leader master found from master 5f24343420014bc2b269b4ed910f9a3a
I20260808 02:22:18.328073 29034 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:22:18.319497 29397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.328354 29397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:18.328466 29397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:18.328595 29397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [term 1 LEADER]: Becoming Leader. State: Replica: 5f24343420014bc2b269b4ed910f9a3a, State: Running, Role: LEADER
I20260808 02:22:18.328754 29397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } }
I20260808 02:22:18.329442 29397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f24343420014bc2b269b4ed910f9a3a. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.329612 29397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:18.333105 29422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:18.334024 29422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:18.346287 29299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [term 1 FOLLOWER]: Refusing update from remote peer 5f24343420014bc2b269b4ed910f9a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:18.346810 29397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:18.348452 29346 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f24343420014bc2b269b4ed910f9a3a. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.348558 29346 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:18.349126 29200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [term 1 FOLLOWER]: Refusing update from remote peer 5f24343420014bc2b269b4ed910f9a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:18.349334 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.329987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.28.177.188:34535)
0808 02:22:18.330030 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.330032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.337321 (+ 7289us) server_negotiation.cc:407] Connection header received
0808 02:22:18.338324 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.338325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.338338 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.338372 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.340940 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.341926 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.344163 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.344548 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.344597 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.346172 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.346174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.346174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.346183 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.347898 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.348361 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.348363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.348363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.348451 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.348681 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.348682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.348683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.348825 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.348934 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.349153 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.349243 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:18.349704 29397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:18.351261 29372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f24343420014bc2b269b4ed910f9a3a. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.351357 29372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:18.352504 29397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.352595 29397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:18.353117 29372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.353207 29372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:18.353557 29425 mvcc.cc:205] Tried to move back new op lower bound from 7316093904262803456 to 7316093904196165632. Current Snapshot: MvccSnapshot[applied={T|T < 7316093904262803456}]
I20260808 02:22:18.354306 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.329198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.28.177.189:37331)
0808 02:22:18.329253 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.329256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.337055 (+ 7799us) server_negotiation.cc:407] Connection header received
0808 02:22:18.337183 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.337184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.337196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.337229 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.338069 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.338827 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.339871 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.340153 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.340201 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.342061 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.342063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.342063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.342072 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.343841 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.344561 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.344562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.344563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.344627 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.352925 (+ 8298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.352928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.352928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.353096 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.353180 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.354121 (+ 941us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.354218 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:22:18.354990 29431 mvcc.cc:205] Tried to move back new op lower bound from 7316093904262803456 to 7316093904196165632. Current Snapshot: MvccSnapshot[applied={T|T < 7316093904262803456}]
I20260808 02:22:18.355401 29417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.355512 29417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:18.355939 29346 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f24343420014bc2b269b4ed910f9a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24343420014bc2b269b4ed910f9a3a" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 34673 } } peers { permanent_uuid: "90c9bf170eaa4f6e92c51d374b4f4055" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 37331 } } peers { permanent_uuid: "fba2559e648245d7bc66b4b704be1fb3" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 34535 } } }
I20260808 02:22:18.356055 29346 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:18.356865 29426 mvcc.cc:205] Tried to move back new op lower bound from 7316093904262803456 to 7316093904196165632. Current Snapshot: MvccSnapshot[applied={T|T < 7316093904262803456}]
I20260808 02:22:18.357501 29422 catalog_manager.cc:1347] Generated new cluster ID: 6009d71cd67b4dfe883cf8ee7c6980a7
I20260808 02:22:18.357553 29422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:18.357970 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:18.329664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.28.177.190:34673)
0808 02:22:18.329712 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.329715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.337118 (+ 7403us) server_negotiation.cc:407] Connection header received
0808 02:22:18.337381 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.337382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.337395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.337429 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.339206 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.340041 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.343401 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:18.343743 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:18.343789 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:18.354691 (+ 10902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:18.354693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:18.354694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:18.354703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:18.356551 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.357024 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.357025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.357026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.357094 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:18.357316 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:18.357317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:18.357317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:18.357473 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:18.357561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:18.357784 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 02:22:18.357878 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:22:18.383409 29422 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:18.384394 29422 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:18.394790 29422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f24343420014bc2b269b4ed910f9a3a: Generated new TSK 0
I20260808 02:22:18.395135 29422 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:22:18.429181 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.427308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.28.177.189:37331)
0808 02:22:18.427359 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.427361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.428378 (+ 1017us) server_negotiation.cc:407] Connection header received
0808 02:22:18.428494 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.428496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.428509 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.428542 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.429101 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39592: BlockingRecv error: recv got EOF from 127.0.0.1:39592 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 02:22:18.430037 29130 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.427762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.28.177.190:34673)
0808 02:22:18.427804 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.427806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.429447 (+ 1641us) server_negotiation.cc:407] Connection header received
0808 02:22:18.429567 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.429568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.429580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.429612 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.429970 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46766: BlockingRecv error: recv got EOF from 127.0.0.1:46766 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 02:22:18.430822 29338 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.428103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.28.177.188:34535)
0808 02:22:18.428156 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.428158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.430276 (+ 2118us) server_negotiation.cc:407] Connection header received
0808 02:22:18.430382 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.430383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.430395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.430425 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.430760 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41092: BlockingRecv error: recv got EOF from 127.0.0.1:41092 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260808 02:22:18.433086 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.432222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.28.177.189:37331)
0808 02:22:18.432262 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.432264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.432488 (+ 224us) server_negotiation.cc:407] Connection header received
0808 02:22:18.432600 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.432601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.432620 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.432652 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.433037 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39600: BlockingRecv error: recv got EOF from 127.0.0.1:39600 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:22:18.436245 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.435435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.28.177.189:37331)
0808 02:22:18.435472 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.435473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.435718 (+ 245us) server_negotiation.cc:407] Connection header received
0808 02:22:18.435816 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.435817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.435828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.435858 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.436194 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39610: BlockingRecv error: recv got EOF from 127.0.0.1:39610 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 02:22:18.440284 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.439409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.28.177.189:37331)
0808 02:22:18.439445 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.439447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.439690 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:22:18.439790 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.439791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.439802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.439853 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.440236 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39620: BlockingRecv error: recv got EOF from 127.0.0.1:39620 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 02:22:18.446630 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.445740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.28.177.189:37331)
0808 02:22:18.445777 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.445779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.446057 (+ 278us) server_negotiation.cc:407] Connection header received
0808 02:22:18.446162 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.446163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.446175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.446206 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.446577 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39624: BlockingRecv error: recv got EOF from 127.0.0.1:39624 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 02:22:18.457085 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.456116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.28.177.189:37331)
0808 02:22:18.456179 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.456181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.456358 (+ 177us) server_negotiation.cc:407] Connection header received
0808 02:22:18.456401 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.456403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.456418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.456510 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.457011 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39640: BlockingRecv error: recv got EOF from 127.0.0.1:39640 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260808 02:22:18.475317 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.474394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.28.177.189:37331)
0808 02:22:18.474449 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.474451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.474633 (+ 182us) server_negotiation.cc:407] Connection header received
0808 02:22:18.474691 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.474693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.474712 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.474801 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.475243 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39646: BlockingRecv error: recv got EOF from 127.0.0.1:39646 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260808 02:22:18.509720 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.508880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.28.177.189:37331)
0808 02:22:18.508957 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.508960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.509079 (+ 119us) server_negotiation.cc:407] Connection header received
0808 02:22:18.509178 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.509180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.509198 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.509266 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.509656 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39648: BlockingRecv error: recv got EOF from 127.0.0.1:39648 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 02:22:18.576041 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.575245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.28.177.189:37331)
0808 02:22:18.575340 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.575342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.575359 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:22:18.575442 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.575443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.575455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.575556 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.575935 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39664: BlockingRecv error: recv got EOF from 127.0.0.1:39664 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 02:22:18.706662 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.705712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.28.177.189:37331)
0808 02:22:18.705797 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.705800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.705832 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:22:18.705920 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.705921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.705935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.705998 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.706607 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39680: BlockingRecv error: recv got EOF from 127.0.0.1:39680 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 02:22:18.965049 29245 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:18.964172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.28.177.189:37331)
0808 02:22:18.964280 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:18.964283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:18.964387 (+ 104us) server_negotiation.cc:407] Connection header received
0808 02:22:18.964491 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:18.964493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:18.964505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:18.964577 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:18.964998 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54740: BlockingRecv error: recv got EOF from 127.0.0.1:54740 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:22:19.247234 29369 catalog_manager.cc:1259] Loaded cluster ID: 6009d71cd67b4dfe883cf8ee7c6980a7
I20260808 02:22:19.247356 29369 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: loading cluster ID for follower catalog manager: success
I20260808 02:22:19.248208 29369 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: acquiring CA information for follower catalog manager: success
I20260808 02:22:19.248579 29369 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fba2559e648245d7bc66b4b704be1fb3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:22:19.290274 29393 catalog_manager.cc:1259] Loaded cluster ID: 6009d71cd67b4dfe883cf8ee7c6980a7
I20260808 02:22:19.290366 29393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: loading cluster ID for follower catalog manager: success
I20260808 02:22:19.291344 29393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: acquiring CA information for follower catalog manager: success
I20260808 02:22:19.291721 29393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90c9bf170eaa4f6e92c51d374b4f4055: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:22:19.481932 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:19.478964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.28.177.189:37331)
0808 02:22:19.479096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:19.479098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:19.479110 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:22:19.479219 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:19.479220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:19.479280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:19.479365 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:19.479580 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.480154 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.481002 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.481276 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.481337 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:19.481506 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:19.481635 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:19.481700 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:19.481770 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:22:19.481792 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260808 02:22:19.487128 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:19.484390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.28.177.188:34535)
0808 02:22:19.484524 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:19.484526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:19.484536 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:19.484578 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:19.484578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:19.484620 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:19.484694 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:19.484872 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.485363 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.486138 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.486332 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.486393 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:19.486680 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:19.486795 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:19.486880 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:19.487015 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 02:22:19.487038 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260808 02:22:19.491974 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:19.489187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.28.177.189:37331)
0808 02:22:19.489233 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:19.489235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:19.489305 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:22:19.489387 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:19.489388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:19.489401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:19.489470 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:19.489626 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.490319 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.491205 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.491424 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.491500 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:19.491665 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:19.491700 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:19.491769 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:19.491867 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:22:19.491895 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:19.492686 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:19.489678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.28.177.188:34535)
0808 02:22:19.489717 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:19.489719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:19.490053 (+ 334us) server_negotiation.cc:407] Connection header received
0808 02:22:19.490111 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:19.490112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:19.490122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:19.490176 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:19.490274 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.490771 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.491845 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.492133 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.492182 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:19.492402 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:19.492440 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:19.492491 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:19.492568 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 02:22:19.492593 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 02:22:19.492898 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:19.489283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.28.177.190:34673)
0808 02:22:19.489420 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:19.489422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:19.489535 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:22:19.489623 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:19.489624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:19.489678 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:19.489764 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:19.489958 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.490697 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.491830 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:19.492122 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:19.492206 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:19.492415 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:19.492535 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:19.492594 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:19.492769 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:22:19.492797 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260808 02:22:19.494423 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29034
I20260808 02:22:19.506966 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29136
I20260808 02:22:19.518532 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29249
2026-08-08T02:22:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3305 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:22:19 dist-test-slave-3h5h krb5kdc[29560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:22:19 dist-test-slave-3h5h krb5kdc[29560](info): set up 2 sockets
Aug 08 02:22:19 dist-test-slave-3h5h krb5kdc[29560](info): commencing operation
krb5kdc: starting...
W20260808 02:22:20.815301 29382 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/test-user.keytab.
Aug 08 02:22:20 dist-test-slave-3h5h krb5kdc[29560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.177.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
Entry for principal kudu/127.28.177.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab.
I20260808 02:22:20.964419 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41649
--webserver_interface=127.28.177.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41649,127.28.177.189:41743,127.28.177.188:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:22:21.051694 29576 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:22:21.051980 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:21.052017 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:21.052062 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:21.053686 29576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:21.053778 29576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:21.053804 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:21.053825 29576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:21.053856 29576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:21.055512 29576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41649,127.28.177.189:41743,127.28.177.188:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.190.keytab
--principal=kudu/127.28.177.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.190:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.177.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:21.055826 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:21.056090 29576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:21.061596 29583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:21.061596 29586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:21.061702 29576 server_base.cc:1034] running on GCE node
W20260808 02:22:21.061635 29584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:21.062100 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:21.062475 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:21.063655 29576 hybrid_clock.cc:648] HybridClock initialized: now 1786155741063631 us; error 39 us; skew 500 ppm
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:21.066778 29576 init.cc:377] Logged in from keytab as kudu/127.28.177.190@KRBTEST.COM (short username kudu)
I20260808 02:22:21.067343 29576 webserver.cc:460] Webserver started at http://127.28.177.190:39579/ using document root <none> and password file <none>
I20260808 02:22:21.067647 29576 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:21.067729 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:21.067898 29576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:21.069339 29576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data/instance:
uuid: "240ae7ec09a14bb8912f5e1dd5cd169d"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "feca24654c3307e347c2db03980a0253"
server_key_iv: "187e14cb7ccb1ff88f3561a29d2d4721"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.069783 29576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal/instance:
uuid: "240ae7ec09a14bb8912f5e1dd5cd169d"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "feca24654c3307e347c2db03980a0253"
server_key_iv: "187e14cb7ccb1ff88f3561a29d2d4721"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.071559 29576 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:22:21.072671 29594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:21.072996 29576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:21.073148 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal
uuid: "240ae7ec09a14bb8912f5e1dd5cd169d"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "feca24654c3307e347c2db03980a0253"
server_key_iv: "187e14cb7ccb1ff88f3561a29d2d4721"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.073237 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:21.089354 29576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:21.091459 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:21.091660 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:21.096451 29576 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.190:41649
I20260808 02:22:21.096553 29672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.190:41649 every 8 connection(s)
I20260808 02:22:21.097021 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/data/info.pb
I20260808 02:22:21.098040 29673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:21.099651 29673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.101224 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29576
I20260808 02:22:21.101372 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-0/wal/instance
I20260808 02:22:21.101606 29382 external_mini_cluster.cc:1428] Setting key d4e00e4f66192dc96de8f129b2202879
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.177.189@KRBTEST.COM; defaulting to no policy
I20260808 02:22:21.112250 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.100373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41649 (local address 127.0.0.1:55304)
0808 02:22:21.100610 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.100655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.101293 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.101724 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.101727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.101999 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.102209 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.102215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.103776 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.103778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.104971 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.104974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.105530 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.105531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.105619 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.106328 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.106339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.108151 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.110333 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.110341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.110343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.110601 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.110899 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.110900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.110901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.110995 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.111476 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.111482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.111552 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.111637 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.111855 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":150,"threads_started":1}
I20260808 02:22:21.112258 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.100416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.28.177.190:41649)
0808 02:22:21.100621 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.100684 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.100747 (+ 63us) server_negotiation.cc:407] Connection header received
0808 02:22:21.101534 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.101537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.101564 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.101690 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.102336 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.103660 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.105066 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.105418 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.105519 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.108265 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.108276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.108277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.108283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.110091 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.110700 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.110704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.110704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.110819 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.111173 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.111174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.111174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.111401 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.111474 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.111736 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.111932 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260808 02:22:21.113334 29673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.113838 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.113660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58236)
0808 02:22:21.113734 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.113798 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.189@KRBTEST.COM" created.
W20260808 02:22:21.114054 29597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
I20260808 02:22:21.114591 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.114480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58246)
0808 02:22:21.114539 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.114566 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 02:22:21.114763 29673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:41743: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab.
Entry for principal kudu/127.28.177.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab.
I20260808 02:22:21.132550 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:41743
--webserver_interface=127.28.177.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab
--principal=kudu/127.28.177.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41649,127.28.177.189:41743,127.28.177.188:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:22:21.164033 29673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } attempt: 1
I20260808 02:22:21.164775 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.164604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58252)
0808 02:22:21.164680 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.164731 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:22:21.165230 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.165116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58254)
0808 02:22:21.165170 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.165205 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260808 02:22:21.165354 29673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:41743: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
W20260808 02:22:21.219549 29684 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:22:21.219861 29684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:21.219880 29684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:21.219901 29684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:21.221208 29684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:21.221253 29684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:21.221268 29684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:21.221280 29684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:21.221293 29684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:21.222859 29684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41649,127.28.177.189:41743,127.28.177.188:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.189.keytab
--principal=kudu/127.28.177.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.189:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.177.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:21.223129 29684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:21.223379 29684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:22:21.228909 29694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:21.229077 29684 server_base.cc:1034] running on GCE node
W20260808 02:22:21.228941 29691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:21.228910 29696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:21.229473 29684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:21.229763 29684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:21.230916 29684 hybrid_clock.cc:648] HybridClock initialized: now 1786155741230904 us; error 31 us; skew 500 ppm
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:21.233505 29673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } attempt: 2
I20260808 02:22:21.233888 29684 init.cc:377] Logged in from keytab as kudu/127.28.177.189@KRBTEST.COM (short username kudu)
I20260808 02:22:21.234205 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.233905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58260)
0808 02:22:21.234116 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.234164 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:22:21.234623 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.234498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58262)
0808 02:22:21.234558 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.234594 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 02:22:21.234632 29684 webserver.cc:460] Webserver started at http://127.28.177.189:46413/ using document root <none> and password file <none>
W20260808 02:22:21.234745 29673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.189:41743: Network error: Client connection negotiation failed: client connection to 127.28.177.189:41743: connect: Connection refused (error 111)
I20260808 02:22:21.234962 29684 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:21.235023 29684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:21.235199 29684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:21.236421 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data/instance:
uuid: "7afb8a07670048c291dff5edf795f438"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "8fc5cb93fbd6140441436f4c049b5bfa"
server_key_iv: "396ed2121cb267b135077f797989a2f1"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.236948 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal/instance:
uuid: "7afb8a07670048c291dff5edf795f438"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "8fc5cb93fbd6140441436f4c049b5bfa"
server_key_iv: "396ed2121cb267b135077f797989a2f1"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.238523 29684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:21.239395 29706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:21.239691 29684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:21.239782 29684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal
uuid: "7afb8a07670048c291dff5edf795f438"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "8fc5cb93fbd6140441436f4c049b5bfa"
server_key_iv: "396ed2121cb267b135077f797989a2f1"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.239838 29684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:21.253558 29684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:21.255975 29684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:21.256102 29684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:21.260864 29684 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.189:41743
I20260808 02:22:21.260944 29786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.189:41743 every 8 connection(s)
I20260808 02:22:21.261363 29684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/data/info.pb
I20260808 02:22:21.262388 29788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:21.264400 29788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.269228 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29684
I20260808 02:22:21.269351 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-1/wal/instance
I20260808 02:22:21.269542 29382 external_mini_cluster.cc:1428] Setting key a5efe1b9d1fc3e2e6b6945662eb171d0
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:21.274606 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.265330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.28.177.190:41649)
0808 02:22:21.265418 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.265420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.265626 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:22:21.266130 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.266132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.266148 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.266229 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.266837 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.267424 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.268462 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.268689 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.268752 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.271426 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.271428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.271428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.271437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.273046 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.273508 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.273510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.273511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.273578 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.273899 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.273900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.273901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.274027 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.274104 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.274414 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.274506 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:22:21.274838 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.265290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41649 (local address 127.0.0.1:55312)
0808 02:22:21.265480 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.265525 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.266013 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.266288 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.266291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.266544 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.266749 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.266752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.267521 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.267522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.268377 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.268380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.268772 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.268773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.268944 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.269599 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.269608 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.271315 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.273139 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.273150 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.273153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.273424 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.273650 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.273651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.273651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.273817 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.274095 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.274103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.274273 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.274395 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.274571 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260808 02:22:21.275458 29788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.177.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.177.188@KRBTEST.COM" created.
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:22:21.285871 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.275795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58272)
0808 02:22:21.275848 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.275860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.275901 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.276196 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.276197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.276365 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.276427 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.276429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.277458 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.277460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.277992 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.277994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.278545 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.278546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.278613 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.279448 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.279455 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.282400 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.284726 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.284727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.284728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.284965 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.285207 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.285208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.285208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.285268 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.285623 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.285624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.285664 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.285717 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.285799 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 02:22:21.286156 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.275863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.28.177.189:41743)
0808 02:22:21.276040 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.276045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.276063 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:22:21.276086 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.276088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.276098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.276181 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.276511 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.277360 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.278086 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.278470 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.278549 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.282509 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.282522 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.282522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.282530 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.284618 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.285051 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.285055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.285055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.285122 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.285332 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.285333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.285333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.285554 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.285655 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.285774 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.285870 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
I20260808 02:22:21.286795 29788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.287214 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.287065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52700)
0808 02:22:21.287120 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.287182 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 02:22:21.287382 29711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
I20260808 02:22:21.287874 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.287752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52706)
0808 02:22:21.287807 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.287841 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 02:22:21.288000 29788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:35723: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.177.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab.
Entry for principal kudu/127.28.177.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab.
I20260808 02:22:21.300777 29382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfrqfAr/build/release/bin/kudu
/tmp/dist-test-taskfrqfAr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:35723
--webserver_interface=127.28.177.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab
--principal=kudu/127.28.177.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.177.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.177.190:41649,127.28.177.189:41743,127.28.177.188:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:22:21.325946 29673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } attempt: 3
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:22:21.333922 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.326389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58282)
0808 02:22:21.326458 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.326477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.326544 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.326749 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.326750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.326901 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.326972 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.326973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.328008 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.328009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.328546 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.328548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.329163 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.329165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.329225 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.329712 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.329717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.330945 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.332798 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.332799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.332800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.333051 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.333285 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.333286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.333286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.333345 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.333660 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.333660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.333714 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.333767 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.333846 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 02:22:21.334025 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.326515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.28.177.189:41743)
0808 02:22:21.326582 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.326585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.326596 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:21.326629 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.326630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.326644 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.326733 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.327059 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.327916 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.328634 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.329048 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.329132 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.331045 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.331046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.331047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.331056 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.332705 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.333133 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.333135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.333135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.333209 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.333414 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.333415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.333416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.333567 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.333641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.333818 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.333914 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:22:21.334406 29673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.334780 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.334671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52714)
0808 02:22:21.334724 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.334758 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:22:21.335179 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.335087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52722)
0808 02:22:21.335128 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.335158 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 02:22:21.335294 29673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:35723: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
I20260808 02:22:21.337226 29788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } attempt: 1
I20260808 02:22:21.337574 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.337462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52724)
0808 02:22:21.337514 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.337542 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 02:22:21.337905 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.337812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52728)
0808 02:22:21.337859 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.337883 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 02:22:21.338002 29788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:35723: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
I20260808 02:22:21.366480 29673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } attempt: 1
I20260808 02:22:21.367029 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.366880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52742)
0808 02:22:21.366954 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.366997 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:22:21.367445 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.367338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52744)
0808 02:22:21.367393 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.367425 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 02:22:21.367542 29673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:35723: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
W20260808 02:22:21.399259 29799 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:22:21.399570 29799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:22:21.399616 29799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:22:21.399667 29799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:22:21.400993 29799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:22:21.401075 29799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:22:21.401101 29799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:22:21.401125 29799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:22:21.401160 29799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:22:21.402897 29799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.177.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.177.190:41649,127.28.177.189:41743,127.28.177.188:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/krb5kdc/kudu_127.28.177.188.keytab
--principal=kudu/127.28.177.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.177.188:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.177.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 01:00:39 UTC on ccycloud.quasar-tvudcf.root.comops.site
build id 4573648
I20260808 02:22:21.403235 29799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:22:21.403513 29799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:22:21.406136 29788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } attempt: 2
I20260808 02:22:21.406610 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.406463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52746)
0808 02:22:21.406528 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.406571 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 02:22:21.407054 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.406965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52762)
0808 02:22:21.407011 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.407039 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 02:22:21.407183 29788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:35723: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
W20260808 02:22:21.409803 29808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:22:21.409803 29809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:21.409970 29799 server_base.cc:1034] running on GCE node
W20260808 02:22:21.409847 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:22:21.410497 29799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:22:21.410897 29799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:22:21.412142 29799 hybrid_clock.cc:648] HybridClock initialized: now 1786155741412130 us; error 40 us; skew 500 ppm
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:22:21.415148 29799 init.cc:377] Logged in from keytab as kudu/127.28.177.188@KRBTEST.COM (short username kudu)
I20260808 02:22:21.415781 29799 webserver.cc:460] Webserver started at http://127.28.177.188:41309/ using document root <none> and password file <none>
I20260808 02:22:21.416070 29799 fs_manager.cc:359] Metadata directory not provided
I20260808 02:22:21.416157 29799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:22:21.416335 29799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:22:21.417874 29799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data/instance:
uuid: "91cce405faae4eaebcb5810e00907cae"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "ac1879093933843348feee42bcad3630"
server_key_iv: "8bf76b13007e0aa90745b906779e8694"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.418334 29799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal/instance:
uuid: "91cce405faae4eaebcb5810e00907cae"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "ac1879093933843348feee42bcad3630"
server_key_iv: "8bf76b13007e0aa90745b906779e8694"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.420089 29799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 02:22:21.421135 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:22:21.421468 29799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:22:21.421635 29799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data,/tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal
uuid: "91cce405faae4eaebcb5810e00907cae"
format_stamp: "Formatted at 2026-08-08 02:22:21 on dist-test-slave-3h5h"
server_key: "ac1879093933843348feee42bcad3630"
server_key_iv: "8bf76b13007e0aa90745b906779e8694"
server_key_version: "encryptionkey@0"
I20260808 02:22:21.421729 29799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:22:21.442755 29673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } attempt: 2
I20260808 02:22:21.443262 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.443107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52770)
0808 02:22:21.443183 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.443224 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:22:21.443631 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.443522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52786)
0808 02:22:21.443573 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.443601 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 02:22:21.443722 29673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.177.188:35723: Network error: Client connection negotiation failed: client connection to 127.28.177.188:35723: connect: Connection refused (error 111)
I20260808 02:22:21.473193 29799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:22:21.475808 29799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:22:21.476001 29799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:22:21.481258 29799 rpc_server.cc:307] RPC server started. Bound to: 127.28.177.188:35723
I20260808 02:22:21.481295 29893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.177.188:35723 every 8 connection(s)
I20260808 02:22:21.481745 29799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/data/info.pb
I20260808 02:22:21.482741 29894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:22:21.484457 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.489151 29382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfrqfAr/build/release/bin/kudu as pid 29799
I20260808 02:22:21.489253 29382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfrqfAr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155444117209-29382-0/minicluster-data/master-2/wal/instance
I20260808 02:22:21.489714 29382 external_mini_cluster.cc:1428] Setting key 863253231319ae1962d4c46896871c1a
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
I20260808 02:22:21.495313 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.485332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.190:41649 (local address 127.0.0.1:55324)
0808 02:22:21.485556 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.485600 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.486175 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.486403 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.486407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.486723 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.486945 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.486949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.487619 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.487620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.488480 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.488482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.488945 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.488947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.489092 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.489776 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.489789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.491697 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.493653 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.493660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.493662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.493902 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.494195 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.494195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.494196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.494366 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.494724 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.494731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.494861 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.494964 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.495089 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":110,"threads_started":1}
I20260808 02:22:21.495386 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.485364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.28.177.190:41649)
0808 02:22:21.485443 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.485446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.485706 (+ 260us) server_negotiation.cc:407] Connection header received
0808 02:22:21.486276 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.486277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.486288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.486345 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.487031 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.487527 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.488578 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.488796 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.488938 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.491796 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.491797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.491798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.491807 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.493567 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.494016 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.494017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.494018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.494084 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.494436 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.494437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.494438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.494628 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.494712 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.495131 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.495239 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:22:21.496138 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:22:21.498351 29788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } attempt: 3
I20260808 02:22:21.500882 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.490377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.28.177.188:35723)
0808 02:22:21.490548 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.490551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.490569 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:22:21.490614 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.490616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.490635 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.490704 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.491406 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.492431 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.493371 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.493876 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.493970 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.497031 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.497041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.497042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.497050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.498783 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.499220 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.499223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.499224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.499291 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.499514 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.499515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.499515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.499720 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.499890 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.500606 (+ 716us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.500709 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.189@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:22:21.505065 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.496420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.189:41743 (local address 127.0.0.1:58286)
0808 02:22:21.496473 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.496485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.496522 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.496744 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.496745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.496927 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.497009 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.497013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.498036 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.498037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.498682 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.498684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.499316 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.499317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.499406 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.499798 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.499803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.501239 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.503711 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.503712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.503713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.503962 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.504242 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.504243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.504243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.504311 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.504664 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.504665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.504714 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.504815 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.504945 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 02:22:21.505064 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.496527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.28.177.189:41743)
0808 02:22:21.496591 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.496593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.496603 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:21.496637 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.496639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.496648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.496712 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.497112 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.497767 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.498765 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.499233 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.499306 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.501328 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.501330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.501330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.501338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.503645 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.504058 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.504059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.504060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.504140 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.504406 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.504407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.504407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.504584 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.504671 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.504884 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.504964 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:22:21.505504 29894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:22:21.508108 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.498610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52806)
0808 02:22:21.498660 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.498670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.498704 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.499077 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.499078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.499235 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.499292 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.499294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.500231 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.500232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.501055 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.501058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.501599 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.501600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.501670 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.502125 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.502130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.503442 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.506917 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.506918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.506918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.507163 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.507365 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.507365 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.507366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.507427 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.507713 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.507714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.507768 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.507855 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.507962 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:22:21.508140 29900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.498784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.28.177.188:35723)
0808 02:22:21.498957 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.498958 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.498965 (+ 7us) server_negotiation.cc:407] Connection header received
0808 02:22:21.498980 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.498981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.498989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.499071 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.499379 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.500109 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.501154 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.501496 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.501597 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.503531 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.503532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.503533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.503540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.506596 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.507231 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.507232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.507232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.507282 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.507502 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.507503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.507503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.507625 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.507702 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.507902 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.507988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.190@KRBTEST.COM
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.188@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:22:21.510663 29788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: Bootstrap starting.
I20260808 02:22:21.511523 29788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:21.511890 29788 log.cc:824] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:21.512310 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.502351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.28.177.190:41649)
0808 02:22:21.502395 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.502397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.502405 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:21.502827 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.502828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.502844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.502883 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.503617 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.504311 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.505435 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.505785 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.505923 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.508786 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.508788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.508788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.508797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.510663 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.511107 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.511109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.511109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.511188 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.511421 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.511423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.511423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.511596 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.511830 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.512104 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.512197 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:21.513022 29788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: No bootstrap required, opened a new log
I20260808 02:22:21.514190 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.505766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52810)
0808 02:22:21.505827 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.505838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.505893 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.506029 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.506030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.506201 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.506237 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.506239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.507091 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.507093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.507974 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.507977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.508763 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.508764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.508819 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.509273 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.509279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.510712 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.512759 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.512760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.512760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.513049 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.513277 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.513278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.513279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.513342 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.513767 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.513767 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.513814 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.513915 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.514065 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 02:22:21.514474 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.505855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.28.177.188:35723)
0808 02:22:21.505893 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.505895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.505903 (+ 8us) server_negotiation.cc:407] Connection header received
0808 02:22:21.505950 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.505951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.505965 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.506015 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.506316 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.506997 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.508066 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.508680 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.508757 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.510806 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.510807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.510807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.510814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.512670 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.513136 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.513137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.513137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.513193 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.513444 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.513444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.513445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.513659 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.513770 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.513940 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.514126 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":18}
I20260808 02:22:21.516005 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: Bootstrap starting.
I20260808 02:22:21.516803 29894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:21.517181 29894 log.cc:824] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:21.518000 29788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.518177 29788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afb8a07670048c291dff5edf795f438, State: Initialized, Role: FOLLOWER
I20260808 02:22:21.518229 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: No bootstrap required, opened a new log
I20260808 02:22:21.518281 29788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.518622 29905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.518687 29788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:21.518707 29905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.177.189@KRBTEST.COM
I20260808 02:22:21.520306 29894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.520530 29894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91cce405faae4eaebcb5810e00907cae, State: Initialized, Role: FOLLOWER
I20260808 02:22:21.520617 29894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
W20260808 02:22:21.520912 29921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:21.520967 29921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:21.520980 29894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:21.522433 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.522547 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:22:21.523339 29938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 02:22:21.523429 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.513448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.28.177.189:41743)
0808 02:22:21.513497 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.513500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.513530 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:22:21.513608 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.513609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.513621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.513693 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.514300 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.514978 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.515969 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.516333 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.516407 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.519316 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.519318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.519318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.519328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.521682 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.522148 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.522150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.522150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.522220 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.522460 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.522461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.522462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.522604 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.522763 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.523258 (+ 495us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.523337 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 02:22:21.523454 29938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:21.521376 29684 master_runner.cc:186] Error getting master registration for 127.28.177.190:41649: OK, instance_id { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" instance_seqno: 1786155741094843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:22:21.524760 29684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:22:21.526157 29799 master_runner.cc:186] Error getting master registration for 127.28.177.190:41649: OK, instance_id { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" instance_seqno: 1786155741094843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:22:21.528048 29799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:22:21.528240 29382 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:22:21.529242 29633 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55336
I20260808 02:22:21.542868 29673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } attempt: 3
Aug 08 02:22:21 dist-test-slave-3h5h krb5kdc[29560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.177.190@KRBTEST.COM for kudu/127.28.177.188@KRBTEST.COM
I20260808 02:22:21.552973 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.543287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.177.188:35723 (local address 127.0.0.1:52812)
0808 02:22:21.543370 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 02:22:21.543384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:22:21.543432 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:22:21.543785 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:22:21.543786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:22:21.543971 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:22:21.544021 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.544023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.545275 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.545277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.546239 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:22:21.546242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.546957 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.546958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:22:21.547025 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.547634 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:22:21.547641 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:22:21.549478 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:22:21.551715 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.551716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.551717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.551925 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.552175 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:22:21.552176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:22:21.552176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:22:21.552247 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:22:21.552565 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:22:21.552567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:22:21.552621 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:22:21.552721 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:22:21.552841 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 02:22:21.552973 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.543562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.28.177.188:35723)
0808 02:22:21.543623 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.543625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.543635 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:21.543666 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.543667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.543678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.543763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.544137 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.545004 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.546384 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.546872 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.546944 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.549590 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.549592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.549592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.549602 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.551633 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.552018 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.552020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.552020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.552090 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.552335 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.552337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.552337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.552499 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.552575 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.552750 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.552860 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:22:21.555686 29673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: Bootstrap starting.
I20260808 02:22:21.556569 29673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: Neither blocks nor log segments found. Creating new log.
I20260808 02:22:21.556955 29673 log.cc:824] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:22:21.557911 29673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: No bootstrap required, opened a new log
I20260808 02:22:21.559505 29673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.559671 29673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240ae7ec09a14bb8912f5e1dd5cd169d, State: Initialized, Role: FOLLOWER
I20260808 02:22:21.559769 29673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.560177 29673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:22:21.560225 29633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:22:21.560278 29633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:21.561332 29633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.561689 29633 leader_election.cc:288] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [CANDIDATE]: Term 1 election: Requested vote from peers 7afb8a07670048c291dff5edf795f438 (127.28.177.189:41743), 91cce405faae4eaebcb5810e00907cae (127.28.177.188:35723)
I20260808 02:22:21.562965 29741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7afb8a07670048c291dff5edf795f438"
I20260808 02:22:21.563151 29741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:21.564334 29741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240ae7ec09a14bb8912f5e1dd5cd169d in term 1.
I20260808 02:22:21.565557 29857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91cce405faae4eaebcb5810e00907cae"
I20260808 02:22:21.565709 29857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:22:21.566779 29857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240ae7ec09a14bb8912f5e1dd5cd169d in term 1.
I20260808 02:22:21.568173 29599 leader_election.cc:302] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240ae7ec09a14bb8912f5e1dd5cd169d, 91cce405faae4eaebcb5810e00907cae; no voters:
I20260808 02:22:21.570785 29576 master_runner.cc:263] No leader master found from master 240ae7ec09a14bb8912f5e1dd5cd169d
I20260808 02:22:21.570839 29576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 02:22:21.562522 29960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:22:21.571049 29960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:22:21.560144 29943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.571288 29943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:21.571398 29943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:22:21.571503 29943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [term 1 LEADER]: Becoming Leader. State: Replica: 240ae7ec09a14bb8912f5e1dd5cd169d, State: Running, Role: LEADER
I20260808 02:22:21.571659 29943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } }
I20260808 02:22:21.572266 29943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240ae7ec09a14bb8912f5e1dd5cd169d. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.572441 29943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:21.574154 29969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:22:21.574983 29969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:22:21.590591 29857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [term 1 FOLLOWER]: Refusing update from remote peer 240ae7ec09a14bb8912f5e1dd5cd169d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:21.591208 29966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:21.592451 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.572936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.28.177.189:41743)
0808 02:22:21.572991 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.572994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.573255 (+ 261us) server_negotiation.cc:407] Connection header received
0808 02:22:21.573443 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.573445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.573460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.573498 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.574534 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.575487 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.576906 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.577266 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.577315 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.588953 (+ 11638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.588955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.588956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.588966 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.590913 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.591398 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.591401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.591401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.591475 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.591712 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.591713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.591714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.591871 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.591994 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.592248 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.592343 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260808 02:22:21.592526 29604 tablet.cc:2369] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:22:21.593760 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240ae7ec09a14bb8912f5e1dd5cd169d. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.593871 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:21.594384 29741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [term 1 FOLLOWER]: Refusing update from remote peer 240ae7ec09a14bb8912f5e1dd5cd169d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:22:21.594722 29966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:22:21.596393 29905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240ae7ec09a14bb8912f5e1dd5cd169d. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.596493 29905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:21.598335 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.573332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.28.177.188:35723)
0808 02:22:21.573375 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.573378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.575988 (+ 2610us) server_negotiation.cc:407] Connection header received
0808 02:22:21.576154 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.576155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.576168 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.576201 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.578427 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.579295 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.582427 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.582771 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.582823 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.587460 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.587462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.587462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.587472 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.589457 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.596712 (+ 7255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.596714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.596715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.596807 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.597521 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.597522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.597523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.597715 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.597828 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.598088 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.598201 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 02:22:21.600884 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:21.573824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.28.177.190:41649)
0808 02:22:21.573875 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.573878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.576972 (+ 3094us) server_negotiation.cc:407] Connection header received
0808 02:22:21.577156 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.577157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.577169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.577201 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.580429 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.581253 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.584602 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:21.585000 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:21.585061 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:21.596964 (+ 11903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:22:21.596966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:22:21.596966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:22:21.596976 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:22:21.599007 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.599820 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.599822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.599822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.599890 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:22:21.600120 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:22:21.600121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:22:21.600122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:22:21.600269 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:22:21.600366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:21.600643 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 02:22:21.600739 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 02:22:21.605767 29966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.605929 29966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:21.606463 29943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.606567 29943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d [sys.catalog]: This master's current role is: LEADER
I20260808 02:22:21.607756 29905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.607851 29905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:21.608489 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240ae7ec09a14bb8912f5e1dd5cd169d" member_type: VOTER last_known_addr { host: "127.28.177.190" port: 41649 } } peers { permanent_uuid: "7afb8a07670048c291dff5edf795f438" member_type: VOTER last_known_addr { host: "127.28.177.189" port: 41743 } } peers { permanent_uuid: "91cce405faae4eaebcb5810e00907cae" member_type: VOTER last_known_addr { host: "127.28.177.188" port: 35723 } } }
I20260808 02:22:21.608673 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:22:21.609536 29976 mvcc.cc:205] Tried to move back new op lower bound from 7316093917540708352 to 7316093917478735872. Current Snapshot: MvccSnapshot[applied={T|T < 7316093917540708352}]
I20260808 02:22:21.609738 29977 mvcc.cc:205] Tried to move back new op lower bound from 7316093917540708352 to 7316093917478735872. Current Snapshot: MvccSnapshot[applied={T|T < 7316093917540708352}]
I20260808 02:22:21.611112 29969 catalog_manager.cc:1347] Generated new cluster ID: 16c9a967abc14ff3a2105db9298cf962
I20260808 02:22:21.611173 29969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:22:21.644116 29969 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:22:21.645123 29969 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:22:21.666584 29969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240ae7ec09a14bb8912f5e1dd5cd169d: Generated new TSK 0
I20260808 02:22:21.666956 29969 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:22:21.695993 29899 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.694013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.28.177.188:35723)
0808 02:22:21.694071 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.694074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.695206 (+ 1132us) server_negotiation.cc:407] Connection header received
0808 02:22:21.695325 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.695327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.695339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.695373 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.695913 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52836: BlockingRecv error: recv got EOF from 127.0.0.1:52836 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260808 02:22:21.697089 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.694494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.28.177.189:41743)
0808 02:22:21.694540 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.694542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.696452 (+ 1910us) server_negotiation.cc:407] Connection header received
0808 02:22:21.696576 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.696577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.696589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.696623 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.697019 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58312: BlockingRecv error: recv got EOF from 127.0.0.1:58312 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 02:22:21.698094 29675 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.694892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.28.177.190:41649)
0808 02:22:21.694938 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.694940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.697516 (+ 2576us) server_negotiation.cc:407] Connection header received
0808 02:22:21.697640 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.697642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.697654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.697687 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.698024 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55362: BlockingRecv error: recv got EOF from 127.0.0.1:55362 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 02:22:21.701221 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.699844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.28.177.189:41743)
0808 02:22:21.699886 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.699888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.700570 (+ 682us) server_negotiation.cc:407] Connection header received
0808 02:22:21.700701 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.700702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.700723 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.700757 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.701165 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58324: BlockingRecv error: recv got EOF from 127.0.0.1:58324 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 02:22:21.704996 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.703899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.28.177.189:41743)
0808 02:22:21.703938 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.703940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.704220 (+ 280us) server_negotiation.cc:407] Connection header received
0808 02:22:21.704428 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.704430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.704475 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.704512 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.704940 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58326: BlockingRecv error: recv got EOF from 127.0.0.1:58326 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 02:22:21.709584 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.708534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.28.177.189:41743)
0808 02:22:21.708592 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.708594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.708952 (+ 358us) server_negotiation.cc:407] Connection header received
0808 02:22:21.709069 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.709071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.709084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.709118 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.709514 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58338: BlockingRecv error: recv got EOF from 127.0.0.1:58338 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 02:22:21.716216 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.715154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.28.177.189:41743)
0808 02:22:21.715199 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.715201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.715496 (+ 295us) server_negotiation.cc:407] Connection header received
0808 02:22:21.715642 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.715643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.715657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.715693 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.716153 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58344: BlockingRecv error: recv got EOF from 127.0.0.1:58344 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 02:22:21.730682 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.729357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.28.177.189:41743)
0808 02:22:21.729418 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.729421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.729822 (+ 401us) server_negotiation.cc:407] Connection header received
0808 02:22:21.729997 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.729999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.730016 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.730056 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.730612 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58360: BlockingRecv error: recv got EOF from 127.0.0.1:58360 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260808 02:22:21.749536 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.748646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.28.177.189:41743)
0808 02:22:21.748702 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.748705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.748827 (+ 122us) server_negotiation.cc:407] Connection header received
0808 02:22:21.748952 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.748953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.748977 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.749062 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.749469 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58362: BlockingRecv error: recv got EOF from 127.0.0.1:58362 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260808 02:22:21.783360 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.782720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.28.177.189:41743)
0808 02:22:21.782786 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.782788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.782844 (+ 56us) server_negotiation.cc:407] Connection header received
0808 02:22:21.782897 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.782898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.782909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.782985 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.783310 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58366: BlockingRecv error: recv got EOF from 127.0.0.1:58366 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260808 02:22:21.849349 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.848657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.28.177.189:41743)
0808 02:22:21.848725 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.848727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.848777 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:22:21.848849 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.848850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.848862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.848928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.849297 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58378: BlockingRecv error: recv got EOF from 127.0.0.1:58378 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 02:22:21.979502 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:21.978703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.28.177.189:41743)
0808 02:22:21.978782 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:21.978784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:21.978795 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:21.978868 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:21.978869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:21.978880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:21.978950 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:21.979438 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58382: BlockingRecv error: recv got EOF from 127.0.0.1:58382 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 02:22:22.237622 29795 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:22:22.236904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.28.177.189:41743)
0808 02:22:22.236975 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:22.236977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:22.236988 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:22:22.237054 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:22.237056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:22.237067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:22.237127 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:22:22.237508 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58390: BlockingRecv error: recv got EOF from 127.0.0.1:58390 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:22:22.521555 29921 catalog_manager.cc:1259] Loaded cluster ID: 16c9a967abc14ff3a2105db9298cf962
I20260808 02:22:22.521675 29921 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: loading cluster ID for follower catalog manager: success
I20260808 02:22:22.522746 29921 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: acquiring CA information for follower catalog manager: success
I20260808 02:22:22.523092 29921 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7afb8a07670048c291dff5edf795f438: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:22:22.524143 29938 catalog_manager.cc:1259] Loaded cluster ID: 16c9a967abc14ff3a2105db9298cf962
I20260808 02:22:22.524206 29938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: loading cluster ID for follower catalog manager: success
I20260808 02:22:22.525251 29938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: acquiring CA information for follower catalog manager: success
I20260808 02:22:22.525600 29938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91cce405faae4eaebcb5810e00907cae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:22:22.754545 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:22.751194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.28.177.189:41743)
0808 02:22:22.751362 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:22.751364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:22.751485 (+ 121us) server_negotiation.cc:407] Connection header received
0808 02:22:22.751583 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:22.751585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:22.751653 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:22.751758 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:22.751945 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.752510 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.753530 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.753824 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.753914 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:22.754121 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:22.754246 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:22.754340 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:22.754386 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 02:22:22.754418 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260808 02:22:22.759714 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:22.756818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.28.177.188:35723)
0808 02:22:22.757028 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:22.757031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:22.757041 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:22:22.757125 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:22.757126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:22.757168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:22.757267 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:22.757386 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.757936 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.758772 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.759006 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.759097 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:22.759292 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:22.759426 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:22.759518 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:22.759575 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 02:22:22.759601 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":254,"threads_started":1}
I20260808 02:22:22.765517 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:22.761840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.28.177.190:41649)
0808 02:22:22.762103 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:22.762105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:22.762212 (+ 107us) server_negotiation.cc:407] Connection header received
0808 02:22:22.762310 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:22.762311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:22.762369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:22.762528 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:22.762681 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.763217 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.764040 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.764253 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.764333 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:22.764554 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:22.764649 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:22.764726 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:22.765223 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 02:22:22.765253 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":57,"threads_started":1}
I20260808 02:22:22.765640 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:22.761891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.28.177.189:41743)
0808 02:22:22.761945 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:22.761948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:22.762151 (+ 203us) server_negotiation.cc:407] Connection header received
0808 02:22:22.762297 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:22.762299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:22.762313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:22.762431 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:22.762599 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.763495 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.764739 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.765047 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.765103 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:22.765316 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:22.765361 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:22.765393 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:22.765529 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 02:22:22.765561 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 02:22:22.765707 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:22:22.761930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.28.177.188:35723)
0808 02:22:22.761973 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 02:22:22.761975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 02:22:22.762410 (+ 435us) server_negotiation.cc:407] Connection header received
0808 02:22:22.762531 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:22:22.762532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:22:22.762545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:22:22.762580 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:22:22.762873 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.763682 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.764802 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:22:22.765113 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:22:22.765182 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:22:22.765385 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:22:22.765421 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:22:22.765474 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:22:22.765590 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 02:22:22.765620 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 02:22:22.767843 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29576
I20260808 02:22:22.780192 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29684
I20260808 02:22:22.794068 29382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfrqfAr/build/release/bin/kudu with pid 29799
2026-08-08T02:22:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3276 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6581 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298692 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST